Knowledge graphs are only useful if they satisfy the requirements of those applications in terms of quality. In this in-use experience paper, we present our approach and tools for supporting the knowledge graph Lifecycle that starts with creation and hosting and continues with the curation and deployment. The curation process enables the maintenance of a knowledge graph, especially in terms of correctness and completeness. We provide process models and evaluation of developed tools with Knowledge Graphs in the tourism domain. We discuss the lessons learned from implementing such an approach in an open and commercial setting in several use cases.
The lifecycle of a knowledge graph comes with two main challenges (1) how to integrate heterogeneous sources in a knowledge graph in a scalable manner (2) how to make them a high-quality resource (e.g., semantically and syntactically correct, no duplicate instances) given the applications in hand.
In this in-use experience paper, we present various tasks of the knowledge graph lifecycle and the tools we developed or adopted to support the knowledge graph lifecycle. The Knowledge Graphs built with this approach are deployed in an open as well as a commercial setting in the tourism domain to support conversational agents. We learned various lessons while implementing our approach about; { orchestration of di erent tasks in the knowledge graph lifecycle { technical and conceptual challenges of dealing with heterogeneous data and distributed actors for knowledge creation
mons License Attribution 4.0 International (CC BY 4.0). { di erent perspectives on quality and knowledge integrity for error detection { challenges of duplication detection con guration during knowledge enrichment
In the remainder of the paper, we rst present the tasks of the knowledge
graph lifecycle (Section 2) and tools developed or adopted to tackle them,
including the use cases for which the knowledge graphs are deployed. Then, we present
the lessons learned from developing and implementing our approach (Section 3).
Finally, we provide concluding remarks and indicators for future work (Section
4). Note that we do not have a dedicated related work section. Naturally, we
bene ted from a plethora of research work while developing our approach. Due
to space restrictions, we do not have a dedicated related work section;
however, we discuss the ones that are directly related to the lifecycle and individual
processes. A comprehensive review can be found in [
There are already proposed methodologies for iterative construction of
Knowledge Graphs from various sources (a recent one is described in [
The lifecycle starts with the creation process that deals with the generation
of semantically annotated data from heterogeneous sources. For this task, we
developed the Importer Tool, which is an ETL tool utilizing RML mappings and
a mapping engine we developed called RocketRML [
The curation process aims to improve the correctness and completeness of a knowledge graph. The rst step is to assess the quality of the knowledge graph. Knowledge quality assessment may involve various dimensions including correctness and completeness6.
Based on the quality score of the correctness and completeness dimensions, the cleaning and enrichment processes can be triggered. The cleaning task involves error detection and correction. We developed the VeriGraph tool that uses integrity constraints implemented with SHACL to detect errors in a knowledge graph. The enrichment process involves detecting duplicate instances in a knowledge graph (or across knowledge graphs) and enrich them with "same as" links.
presented in this paper. An early demo can be seen at https://qat.semantify.it Knowledge Curation
VeriGraph Knowledge Creation Importer
and semantify.it Knowledge Hosting
GraphDB Knowledge Assessment
Duplication Detection
as a Service Knowledge Cleaning
Knowledge Enrichment Conversational Agents
and Open and Commercial Knowledge Deployment
Knowledge Graphs The property values of the linked instances may need to be fused afterward, which may require another cleaning process to identify violations of integrity constraints (e.g., a property with one maximum cardinality may have multiple values after the fusion.). The curated knowledge graphs are nally deployed to be consumed by various applications. These processes continue iteratively to constitute the lifecycle. In the remainder of the section, we focus on the creation, cleaning, and enrichment tasks and present the tools we employed to tackle these tasks including some design decisions. Finally, knowledge deployment will be presented in the form of use cases. 2.1
Knowledge creation, as de ned in [
In principle knowledge, creation can be done manually, for example via a GUI, through mappings from (semi-)structured sources or semi-automatically from unstructured sources. In our use cases, the majority of the knowledge is created via declarative mappings from hierarchical data sources. In the context of our work, the data to be integrated into the knowledge graph was obtained from di erent service providers in di erent formats, typically JSON or XML. The mapping was then de ned from those sources to schema.org.
The process model for knowledge creation via mappings is the following: We rst collect raw data from di erent service providers via web services. Each object retrieved is mapped to schema.org to create an instance of a schema.org type with its property values assertions. Alternatively, data providers can provide RDF data directly. The generated or acquired RDF data is then enriched with provenance information based on PROV-O7.
Besides the technical challenge of creating mapping les, there is also a
conceptual one that involves the selection of types and properties for the mapping.
It is even more challenging when the actors creating the mappings and
selecting the types and properties are not the same. We developed an approach for
creating domain-speci c patterns of schema.org and its extensions[
We implemented the creation process in the Importer tool. The tool allows registration of new sources including all the access information and their RML mapping les. The timing and frequency of the mappings can be speci ed with cron strings. The Importer has Apache NiFi9 in its core to manage the entire data ow from accessing raw data to storing it in a triplestore. Apache NiFi is a data ow management tool that o ers load balancing, bu ering and guaranteed delivery. The actual mapping is executed via an external RocketRML instance, a scalable RML mapper implemented with NodeJS.
RocketRML currently supports JSON, XML, and CSV formats and adopts optimization techniques like JOIN path memoization for high-performance. It supports various JSON-Path and XPath implementations allowing users to access extra features like backward traversal in a JSON le with JSON-Path Plus10. It also supports function mappings which are frequently used for transforming property values and distinguishing between di erent subtypes of schema.org type during the mapping (e.g., di erent types of events can be dynamically mapped with a single mapping). Listing 1 shows an excerpt from a mapping supported by RocketRML11. The mapping le creates schema:LocalBusiness instances from various touristic regions' data in Tyrol. Note that the getType function returns the suitable subtype of schema:LocalBusiness (e.g., schema:Store, schema:Library) for a given instance. The relationship between a local business and its opening hours is given by the nested structure of XML tags and not
#prefixes sources: ... mappings: acc: sources:
- acc s: ml:$(@Id) po: - [a, {function: myfunc:getType, parameters: ,! ["$(Details/Topics/Topic/@Id)"]}] - [schema:name, "$(Details/Names/Translation[@Language= de ])", ,! de~lang] - [schema:name, "$(Details/Names/Translation[@Language= en ])", ,! en~lang] ... - [schema:openingHoursSpecification, {mapping: hours, join: [@Id, ,! ../../../../@Id]}] ...
Listing 1: An excerpt from an RML mapping with YARRML syntax for local businesses in touristic regions by primary-foreign key relationships. Therefore, the mapping uses a backward traversal from opening hours object to the local businesses' Id eld to join them properly.
The RML mapper used in the Importer, RocketRML, can map 25K triples
per second on average. However, the overhead caused by sending queries to the
GraphDB instance over HTTP harms the overall import process. A more detailed
explanation of knowledge creation via mappings and a detailed evaluation of the
importer tool can be found in [
Knowledge Curation is a process for assessing and improving a knowledge graph
in various dimensions, especially correctness and completeness (see also
"knowledge re nement" [
Knowledge Cleaning Knowledge Cleaning is a process that aims to improve the correctness of a knowledge graph. It consists of (a) error detection, the task for identifying the erroneous type and property value assertions, and (b) error correction, xing the identi ed statements. We focused on the former, particularly the veri cation task where the knowledge graph is checked against a speci cation such as integrity constraints.
Our approach is based on verifying the instances in a knowledge graph against the domain-speci c patterns of schema.org. These patterns are expressed with a subset of SHACL 12. For such an approach we considered various SHACL veri ers but none of them covered our needs properly. RDFUnit was the closest tool to satisfy our requirements as it is triplestore-independent and can work directly on SPARQL endpoints without loading a data dump to the memory. However, we had issues with large Knowledge Graphs in some of our use cases (particularly Tyrolean Tourism Knowledge Graph) as the veri cation never ended after 10M triples (see below for evaluation).
For detecting errors in a knowledge graph, we conceptualized and developed
a veri er that checks whether a particular subset of a knowledge graph ts the
domain-speci c pattern. Figure 2 shows the process model.
The rst step loads a domain-speci c pattern that comprises the shapes
graph for veri cation. Then the veri cation process is split into two lines: The
rst line of veri cation retrieves the URIs of the instances that match the target
speci cation and adds them to a veri cation queue. Then, for each URI in the
12 Details of the subset can be found in [
We implemented our error detection approach in the VeriGraph tool13. The tool has been implemented in Javascript and available with an open license. It can be con gured to run on any knowledge graph that provides a SPARQL endpoint. In our experience with many SHACL veri ers, we realized there are generally two main issues in practice: (1) operating in-memory, which causes insu cient memory problems with large data graphs (2) SPARQL endpoints are not always reliable for frequent queries that return high-volume results. The rst issue we address with a caching mechanism. The data graphs are cached on the disk and only loaded to the memory when they are needed for veri cation. The second issue is addressed by both indexing and the caching mechanism. Indexing the URIs and querying their data graphs one-by-one reduces the size of the data graph returned by a single query. Each constraint component de ned by the property shapes is checked by graph-traversal in the memory. This reduces the number of SPARQL queries running against an endpoint for veri cation. Additional to the typical SHACL veri cation report, the VeriGraph tool provides metadata about the veri cation process (e.g., duration, number of violations found).
We evaluated the VeriGraph on several subgraphs of Tyrolean Tourism Knowledge Graph with an increasing number of triples, starting from 100K up to 1B14. Each knowledge graph contains instances of types like Event, Hotel, HotelRoom, Person, and Product. The evaluation has been conducted on a server with an Intel Core i9-9900K Octa-Core 3.60GHz processor, 64GB RAM, and 2TB SSD. We compared our implementation with the SHACL veri er of AllegroGraph (via agTool)15, RDFUnit, built-in SHACL veri er of Stardog, and TopBraid SHACL API16. The instances were veri ed against a set of constraints with di erent target speci cations17, except for one constraint for Stardog, due to a non-supported constraint type (a property-pair constraint). Figure 3 shows the veri cation time in relation to knowledge graph size. While each tool detects about the same num13 https://github.com/semantifyit/VeriGraph 14 http://dataset.sti2.at/datasets/ 15 https://franz.com/agraph/support/documentation/6.6.0/shacl.html 16 https://github.com/TopQuadrant/shacl 17 https://github.com/semantifyit/VeriGraph/blob/master/constraints/constraints.ttl ber of violations for the same size, in terms of time spent, VeriGraph stands out as the size grows. It is the only one that can nish veri cation on a knowledge graph with 1B triples. In smaller Knowledge Graphs, VeriGraph is behind the tools either working completely in-memory (TopBraid) or on their triplestores natively (AllegroGraph, Stardog) for smaller datasets. For RDFUnit and VeriGraph connecting to generic SPARQL endpoints of triplestores create an overhead. Our initial investigations showed that RDFUnit's performance is affected primarily not by the size but the number of violations found. This could be because of generating one SPARQL query for each constraint component and overhead caused by processing the results of these queries to create veri cation reports. Nevertheless, VeriGraph is not a complete SHACL veri er as it only recognizes a subset of SHACL (e.g. only class-targets are allowed). However, it appears to be a feasible choice for our use cases.
()s 105 e m i t ion 103 t a c i r e lv 101 l a r e v O 10 1 10 1 108
Knowledge Enrichment Knowledge Enrichment is a process that aims to
improve the completeness of a knowledge graph [
Mapping
Indexing
Pre-filtering Property-value standardization
Instance comparison
Decision model application Indexing Phase Duplicate Detection Phase
We developed a highly con gurable service-oriented approach to the
duplicate detection problem that allows linking duplicate instances in a knowledge
graph or from external Knowledge Graphs [
The duplication detection problem is almost as old as computer science itself
and there have been a plethora of approaches to tackle it. We also examined
various tools such as Duke[
The "likeness" is calculated with TF-IDF. Putting a high threshold for the number of matching terms may harm the recall of the overall approach. See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mltquery.html . thresholds but also indexing and pre- ltering steps. We provide the possibility to use various algorithms such as time-constrained brute force, hill climbing, localized brute force, and genetic algorithms with random mutations for each parameter. The con guration learning can be adjusted towards optimizing for not only F-score but also recall or precision.
We implemented our approach in a tool called Duplicate Detection as a Service (DDaaS). The tool consists of multiple services for di erent tasks. These services are orchestrated via REST. The decision in the favor of a service-oriented architecture is to facilitate easy replacement of individual components and their independent development.
To evaluate the approach, we compared it with three other tools, Duke, LIMES and Silk, all of which also in uenced the development of DDaaS. We compared the tools over two datasets (Restaurants20 and SPIMBENCH21). The results are displayed in Table 1.
Restaurants
Tool F1-Score Precision Recall DDaaS 0.76 1.00 0.61 Duke 0.77 1.00 0.62 LIMES 0.80 0.86 0.74
Silk 0.40 0.79 0.27
SPIMBENCH
Tool F1-Score Precision Recall DDaaS 0.85 0.98 0.76 Duke 0.09 0.05 0.75 LIMES 0.72 0.88 0.61
Silk 0.62 0.77 0.53
We ran every tool the most automated way possible and results indicate that DDaaS is at least on par with the other tools. The SPIMBENCH results are particularly interesting here, while the results for the Restaurants dataset are more balanced. The main di erences between these datasets lie in the number of properties and their completeness. While the Restaurant dataset is a perfect toy dataset, the instances in the SPIMBENCH dataset are very sparse with regards to property values on many instances. Duke's purely genetic approach to learning a con guration for this particular dataset seems to be awed as it will keep retrying to use properties that are not even available for most instances. DDaaS' approach to learning a con guration is a composition of di erent approaches which includes a genetic approach but also less randomized approaches. We call this composition the learning strategy. Since every single element of this strategy (i.e., con gurable pieces) can be con gured to aim to optimize one of the three measures (precision/recall/F1), it achieved a good performance even on such an incomplete dataset. Further evaluation on Tyrolean Tourism Knowledge Graph is ongoing. 20 https://www.cs.utexas.edu/users/ml/riddle/ 21 https://project-hobbit.eu/challenges/om2020/ 2.3
"The proof of the pudding is in the eating." A knowledge graph is only as valuable as the applications it enables. Therefore, the knowledge deployment task deals with the applications that are powered by a knowledge graph. In this section, we present three use cases where this lifecycle-based approach is being implemented to build knowledge graphs to power various applications.
Graph is the reference project to implement the presented lifecycle-based approach. A "knowledge management tool" is currently being implemented to manage creation, hosting, curation, and deployment. The knowledge graph will integrate tourism-related data from the tourism marketing organizations of 16 federal states and many other external sources. The knowledge graph management tool will o er features like provenance tracking, machine-understandable licenses, and various types of visualizations as well as APIs for programmatically accessing to the knowledge graph. The knowledge graph will follow domainspeci c patterns of schema.org and its extension developed by the Open Data Travel Alliance which is an organization that brings tourism experts from Austria, Germany, Italy, and Switzerland to create schemas for tourism knowledge graphs. These patterns are published in human-readable form and as SHACL shapes to help the data providers from the touristic marketing organizations to create RDF data via declarative mappings. Same patterns will be used to verify the incoming data.
The project started in December 2020 and will run for 2 years, with an initial prototype in May 202122. It will be open to application developers from di erent domains (e.g., mobility service providers, online travel agencies) and aim to foster the development of intelligent applications that improve e-tourism processes.
Tourism Knowledge Graph23, which contains more than 12B statements. It is populated with data from 11 di erent sources (mainly Destination Management Organizations from di erent regions in Tyrol) and updated daily. The presented knowledge creation process is applied fully in this use case and knowledge cleaning processes are running on demand. The knowledge coming from di erent sources is organized in named graphs. The named graphs imported from the same source on di erent time points are linked with each other via the provenance information, which allows applications like time series analysis on frequently changing data (e.g., accommodation prices, weather measurements). Onlim Conversational Agents Onlim GmbH deploys knowledge graphs developed with the presented approach commercially. Onlim uses their knowledge 22 See the project timeline online. - https://open-data-germany.org/projektstand/ 23 http://tirol.kg graph to power their conversational agents in di erent domains such as tourism, education, energy, and nance. Most notably in the tourism domain, they provide about 20 conversational agents. These agents are typically goal-oriented dialog systems (GDS) that help users to achieve their goals via conversations. In the case of Onlim, a knowledge graph powers a GDS in two di erent ways: { Providing entities for annotating user utterances to train Natural Language
Understanding (NLU) models. { Serving as a knowledge source to provide the knowledge needed for a task at hand
Onlim uses state-of-the-art GDS development frameworks such as DialogFlow24 and RASA25 to streamline the conversational aspects of a dialog system such as NLU, dialog management and NLG. Such frameworks work with intents, structures that represent the user goals a GDS supports. The frameworks use supervised machine learning to classify utterances to intents. They use Knowledge Graphs to create annotated utterances for each intent to help the machine learning models classify incoming utterances to the correct intents. An intent is then mapped to a SPARQL query and the user's question is answered based on the data provided by the knowledge graph (e.g., accommodation, events, infrastructure). The lifecycle explained throughout the paper ensures that the answers returned a high quality (e.g., resolved duplicate instances, correct property values). An example of such a GDS can be found online26. 3
In this section, we discuss our lessons learned from implementing the presented lifecyle, from the perspective of the overall approach, and creation and curation processes with references to the use cases above. 3.1
cycle involves many tasks and each task has been addressed in the literature
with a vast number of methods and tools. We realized however there is an
important gap while implementing the lifecycle which is a tool that can provide
periodic and on-demand actuation of various processes and their orchestration.
An open and decoupled architecture can facilitate relatively painless integration
of di erent steps in the lifecycle. The interoperability of di erent tools targeting
di erent tasks remains an interesting research and engineering challenge. Here
we can take a page from the book of the question-answering systems community
as they have been proposing an open architecture to increase the reusability
and interoperability of the tools targeting di erent steps of question-answering
system development [
Knowledge Graphs from heterogeneous sources scale well with declarative mappings. However, the data received from real-world IT solution providers are not always ideal. For instance, in the Tyrolean Tourism Knowledge Graph use case, we frequently encountered data sources that do not provide any elds to join two logical sources (e.g., events and their organizers) but the relationship is speci ed by nested structures. We worked around this by extending the existing JSONPath and XPath27 implementations with a ~PATH term which represents the absolute path of a value in the JSON or XML tree, which is suitable for joining nested structures. Moreover, various source-speci c cleaning steps are involved in many cases, which hinders scalable development. Here again community e orts like KGC CG can be bene cial for identifying common challenges in declarative mappings and addressing them within the existing tools and approaches.
mappings are clear in terms of exibility and reusability, however, the conceptual and social challenges still stand. The domain experts must de ne the domain by identifying relevant types, properties, and constraints and communicate them to the developers and mapping creators. This is particularly challenging when these actors are distributed across di erent organizations, as it is in the German Tourism Knowledge Graph, where semantically annotated data comes from at least 16 di erent organizations to be integrated into a single knowledge graph. We experienced that simple human- and machine-understandable patterns of schema.org and its extensions published by domain experts improve the knowledge creation process signi cantly. 3.3
ence we had with Tyrolean and German Tourism Knowledge Graph use cases is 27 Newer XPath implementations already have similar functions. that the di erent instances of the same type may have di erent expected shapes. For instance, a generic Organization shape may require schema:vatID property however for a schema:Organization instance that is the value of organizer property of an event only the name property may be interesting. A SHACL shape that targets the Organization type would verify both Organization instances, which is not the intended behavior. To address this, we see domain-speci c patterns as types with local properties and ranges. This means the relevant shape of each instance has to be asserted on that instance. Then the veri cation turns into instance checking under Closed-World Assumption.
ing cleaning process For commercial applications of knowledge graphs such
as the Onlim Conversational Agents, maintaining constant knowledge integrity
is crucial, especially in domains like energy and nance. No matter how e cient
the constraint checking process is conducted, the processing time may go out of
acceptable limits as the knowledge graph size and constraint complexity increase.
Distinguishing between constraints involving only metadata and constraints
involving property values can help the scalability of constraint checking as the
former can be done over the mapping les which has proven to be e cient[
In this paper, we presented our experience with implementing a knowledge graph lifecycle including creation, curation, and deployment. We presented the tasks in the knowledge graph lifecycle and employed a set of tools for many of those tasks. The current implementation is lacking a proper tool to orchestrate the lifecycle but it is currently being developed in use cases like the German Tourism Knowledge Graph. There are still some tasks such as validating knowledge graphs against the real world, fusing linked instances, and automating error correction that requires further research. Moreover, the maturity of our tools is at a di erent stage, however, they are actively being developed by industrial adopters such as Onlim. We provided an evaluation of di erent tools supporting the lifecycle individually. Their real evaluation will be in the next couple of years as the developed approaches and tools are continuously being tested in the knowledge graphs and applications of Onlim.
As a knowledge graph gets bigger and supports more applications, it may come to a point that the curation process may be infeasible, both due to the size of the knowledge graph and changing contexts (e.g., di erent applications and customers may have a di erent set of constraints and rules). Therefore, our further research will focus also on building a layer on top of knowledge graphs that enables applications to work on small subsets of knowledge graphs with di erent con gurations for curation which will allow the customization of the knowledge graph for di erent application contexts.
This work has been partially funded by the industrial research project MindLab28. 28 https://mindlab.ai/