The LDIF - Linked Data Integration Framework can be used within Linked Data applications to translate heterogeneous data from the Web of Linked Data into a clean local target representation while keeping track of data provenance. LDIF provides an expressive mapping language for translating data from the various vocabularies that are used on the Web into a consistent, local target vocabulary. LDIF includes an identity resolution component which discovers URI aliases in the input data and replaces them with a single target URI based on user-provided matching heuristics. For provenance tracking, the LDIF framework employs the Named Graphs data model. This paper describes the architecture of the LDIF framework and presents a performance evaluation of a life science use case.
The Web of Linked Data grows rapidly but the development of Linked Data
applications is still cumbersome due to the heterogeneity of the Web of Linked
Data [
In contrast to the heterogeneity of the Web, it is bene cial in the application context to have all data describing one class of entities being represented using the same vocabulary. Instead of being confronted with URI aliases which refer to data that might or might not describe the same entity, Linked Data applications would prefer all triples describing the same entity to have the same subject URI as this eases many application tasks including querying, aggregation and visualization.
In order to ease using Web data in the application context, it is thus advisable to translate data to a single target vocabulary (vocabulary mapping) and to replace URI aliases with a single target URI on the client side (identity resolution), before doing any more sophisticated processing.
There are various open source tools available that help application developers with either data translation or identity resolution. But to date, there are hardly any integrated frameworks available that cover both tasks. With LDIF, we try to ll this gap and provide an open-source Linked Data integration framework that provides for data translation and identity resolution while keeping track of data provenance.
Figure 1 shows the schematic architecture of a Linked Data application that
implements the crawling/data warehousing pattern [
Application Layer Data Access, Integration and Storage Layer Web of Data
Application Code Web Data Access Module
HTTP
LDIF Vocabulary Mapping Module
Identity Resolution Module
SPARQL or RDF API Quality Evaluation Module
Integrated Web Data Publication Layer
HTTP
HTTP
HTTP LD Wrapper
LD Wrapper Database A
Database B
RDFa CMS
The LDIF framework is implemented in Scala and can be downloaded from the project website1 under the terms of the Apache Software License. In the following, we explain the architecture of the LDIF framework and present a performance evaluation along the example of a life science use case.
The LDIF framework consists of a runtime environment and a set of pluggable modules. The runtime environment manages the data ows between the modules. The pluggable modules are organized as data access components, data transformation components and data output components. So far, we have implemented the following modules: 2.1
The current version of LDIF expects input data to be represented as Named Graphs and be stored in N-Quads format. The graph URI is used for provenance tracking. Provenance meta-information describing the graphs can be provided as part of the input data within a speci c provenance graph. The name of this provenance graph can be set in the LDIF con guration le. LDIF does not make any assumptions about the provenance vocabulary that is used to describe the graphs, meaning that you can use your provenance vocabulary of choice. 2.2
LDIF employs the R2R Framework [
LDIF employs the Silk Link Discovery Framework [
Silk uses a novel blocking approach which removes de nite non-duplicates early in the matching process, thereby signi cantly increasing its e ciency. For each set of duplicates which have been identi ed by Silk, LDIF replaces all URI aliases with a single target URI within the output data. In addition, it adds owl:sameAs links pointing at the original URIs, which makes it possible for applications to refer back to the original data sources on the Web. If the LDIF input data already contains owl:sameAs links, the referenced URIs are normalized accordingly. 2.4
The N-Quads writer dumps the nal output of the integration work ow into a single N-Quads le. This le contains the translated versions of all graphs from the input graph set as well as the contents of the provenance graph. 2.5
The runtime environment manages the data ow between the modules and the caching of the intermediate results. In order to parallelize processing, data is partitioned into entities prior to supplying it to a transformation module. An entity represents a Web resource together with all data that is required by a transformation module to process this resource. Entities consist of one or more graph paths and include a provenance URI for each node. Each transformation module speci es which paths should be included into the entities it processes. By splitting the data set into ne-grained entities, LDIF is able to parallelize the workload on machines with multiple cores. In the next release, it will allow the workload to be parallelized on multiple machines using Hadoop. 3
We evaluated the performance of LDIF using two life science data sets: KEGG GENES2, a collection of gene catalogs generated from publicly available resources, and UniProt3, a data set containing protein sequence, genes and functions.
We de ned R2R mappings for translating genes, diseases and pathways from KEGG GENES and genes from UniProt into a proprietary target vocabulary4. The mappings employ complex structural transformations. The prevalent value transformations rely on regular expressions, e.g. for extracting an integer value from a URI, and modify the target data types. We de ned Silk linkage rules for identifying equivalent genes in both datasets. For the benchmark, we generated subsets of both data sources together amounting to 25 million, 50 million, and 2 http://www.genome.jp/kegg/genes.html 3 http://www.uniprot.org/ 4 http://www4.wiwiss.fu-berlin.de/bizer/ldif/resources/Wiki.owl 100 million quads. The R2R mappings, Silk linkage rules as well as the evaluation data sets can be downloaded from the LDIF website.
We ran the performance tests on a machine with an Intel i7 950, 3.07GHz (4 cores) processor and 24GB of memory out of which we assigned 20GB to LDIF.
Table 1 summarizes the LDIF runtimes for the di erent data set sizes. The overall runtime is split according to the di erent processing steps of the integration process.
Table 2 provides statistics about the data integration process. The original number of input quads decreases in the process as LDIF was con gured to discards input quads which are irrelevant for the de ned mappings, and therefore can not be translated into the target vocabulary. The number decreases again after the actual translation, as the input data uses more verbose vocabularies and as multiple quads from the input data are thus combined into single quads in the target vocabulary. We are aware of two other Linked Data integration frameworks that also provide for data translation and identity resolution: The ALOE - Assisted Linked Data Consumption framework5 developed at the Universitat Leipzig and the
Information Workbench6 [
Over the next months, we will extend LDIF along the following lines:
This work was supported in part by Vulcan Inc. as part of its Project Halo (www.projecthalo.com) and by the EU FP7 project LOD2 - Creating Knowledge out of Interlinked Data (http://lod2.eu/, Ref. No. 257943).