The Web of Linked Data is built upon the idea that data items on the Web are connected by RDF links. Sadly, the reality on the Web shows that Linked Data sources set some RDF links pointing at data items in related data sources, but they clearly do not set RDF links to all data sources that provide related data. In this paper, we present Silk Server, an identity resolution component, which can be used within Linked Data application architectures to augment Web data with additional RDF links. Silk Server is designed to be used with an incoming stream of RDF instances, produced for example by a Linked Data crawler. Silk Server matches the RDF descriptions of incoming instances against a local set of known instances and discovers missing links between them. Based on this assessment, an application can store data about newly discovered instances in its repository or fuse data that is already known about an entity with additional data about the entity from the Web. Afterwards, we report on the results of an experiment in which Silk Server was used to generate RDF links between authors and publications from the Semantic Web Dog Food Corpus and a stream of FOAF pro les that were crawled from the Web.
The Web of Linked Data [
DataSets/LinkStatistics between data items from di erent data sources thus might want to augment Web data with additional links before using it in the application context.
In order to tackle this problem, we provide the Silk Link Discovery
Framework [
Silk Single Machine is used to generate RDF links between two datasets on a single machine.
Silk MapReduce is based on Hadoop and enables Silk to scale out to very big datasets by distributing the link generation to multiple machines. Silk Server can be used as an identity resolution component within applications that consume Linked Data from the Web.
This paper is focused on Silk Server which has been recently added as a new component to the Silk Link Discovery Framework. Silk Single Machine and Silk MapReduce are described on the Silk homepage2.
Silk Server is designed to be used with an incoming stream of RDF instances, produced for example by a Linked Data crawler such as LDSpider3. Silk Server matches incoming instances against a local set of known instances and discovers missing links between them. Incoming instances which do not match a known instance are added to the local set of instances continuously. Based on this assessment, an application can store data about newly discovered instances in its repository or fuse data that is already known about an entity with additional data about the entity from the Web.
The main features of the Silk Server are:
It runs as an HTTP server and o ers a REST interface [
It is high-performing by holding the data about all known instances in an inmemory cache, which is updated as soon as new instances are discovered. In addition, the performance can be further enhanced using a blocking feature.
The paper is structured as follows: Section 2 explains the role Silk Server can
play within Linked Data application architectures. In Section 3, the architecture
and work ow of the Silk Server are presented. Section 4 reports on the results
of an experiment in which Silk Server was used to generate RDF links between
the data about authors and publications from the Semantic Web Dog Food
Corpus [
This section discusses the role of Silk Server within Linked Data application
architectures. Figure 1 gives an overview of the architecture of a fully- edged
Linked Data application which operates on top of the public Web of Linked
Data [
All data that is published on the Web according to the Linked Data principles
becomes part of a giant global graph - the Web of Linked Data. This logical graph
is depicted in the Web of Linked Data layer in Figure 1. Applications that utilize
this graph might implement the modules (or a subset of the modules) depicted in
the Data Access, Integration and Storage Layer. In the following, we will describe
the functionality of the di erent modules.
1. Web Data Access Module The basic means to access Linked Data on the
Web is to dereference HTTP URIs into RDF descriptions and to discover
additional data by traversing RDF links. Such link traversal can for instance
be implemented using readily available Linked Data crawlers such as
LDspider. In addition, the data access module might download RDF data set
dumps or utilize SPARQL endpoints (for an overview about SPARQL-based
distributed query architectures please refer to [
Web data is stored in a repository together with provenance information to
be used by the application layer. A commonly used model for representing
Web data together with provenance information are Named Graphs [
Silk Server is an identity resolution component that can be used within Linked Data application architectures. It runs as an HTTP server and matches instances of an incoming RDF stream against a local set of known instances based on userprovided link speci cations. In the following, we will describe the architecture of the Silk Server as well as the general linking work ow. 3.1
The Silk Server is composed of the following three layers:
The in-memory instance cache builds the bottom layer which holds all known instances and keeps track of newly discovered instances. For each instance, the values of all relevant properties which are later required for the comparison are stored. As soon as a new instance is discovered, it is added to the instance cache. This enables the server to generate links to the newly discovered instance in future requests. Currently, the instance cache is held in memory, but can be replaced by a persistent cache in future versions of Silk. The current implementation of the instance cache can t approximately 10 million instances into 8GB of main memory.
The Silk Linking Engine generates the links based on a set of link speci cations and forms the central part of Silk Server. The details of the link generation process are covered in Section 3.3.
The REST interface enables applications to commit newly discovered resources and receive the generated links. New resources are accepted through an HTTP POST request using one of the supported RDF serialization formats, such as RDF/XML or N-Triples. The response contains all generated links optionally including statements declaring unknown instances i.e. instances for which no link could be generated. The server can process multiple requests in parallel. 3.2
6 http://www.w3.org/2005/Incubator/prov/ The Silk Server work ow is divided into 2 phases:
In the Setup phase the server loads all data sets which are speci ed by the user-provided link speci cations. For each link speci cation, one instance cache is used to hold the part of the data that is later required for matching instances.
The Service phase starts as soon as all data sets have been loaded. If an application discovers new instances on the Web, it issues a request to the server containing the newly found data. The request may contain multiple instances with di erent types. On receiving the request, the Server matches the given instances with its link speci cations. If a link speci cation can be applied to a speci c instance, the server forwards it to the Silk Linking Engine. The Silk Linking Engine generates links for the given instances based on the corresponding link speci cations as described in Section 3.3.
The generated links are processed by the server to nd the set of instances which are not matched by any known instance. The instance cache is updated with the set of unmatched instances. Thus, in future request the server will also generate links to the newly found unmatched instances.
After the update has been completed, the generated links along with statements containg the unmatched instances are returned. 3.3
When receiving new instances to be matched, the Silk Linking Engine generates new buckets consisting of a provided instance and a set of instances from the cache. Each bucket is processed in 3 subsequent phases:
The optional Blocking phase partitions the incoming buckets into clusters. Since comparing every source resource to every single target resource results in a number of n m comparisons (n being the number of source resources, m the number of target resources), blocking can be used to reduce the number of comparisons. Blocking partitions similar data items into clusters limiting the comparisons to items in the same cluster. For example, given a set of books to be compared, in order to reduce the number of comparisons, one could block the books by publisher. In this case only books from the same publisher will be compared.
The Link Generation phase reads the incoming buckets and computes a similarity value for each pair of instances. The incoming data items, which might be allocated to a cluster by the preceding blocking phase, are written to an internal cache. From the cache, pairs of data items are generated. If blocking is disabled, this will generate the complete cartesian product of the two data sets. If blocking is enabled, only data items from the same cluster are compared. For each pair of data items, the link condition is evaluated, which computes a similarity value between 0 and 1. Each pair generates a preliminary link with a con dence according to the similarity of the source and target data item.
The Filtering phase lters the incoming links in two stages: In the rst stage, all links with a lower con dence than the user-de ned threshold are removed. In the second stage, all links which originate from the same subject are grouped together. If a limit is de ned on the number of links per subject, only the links with the highest con dence are forwarded to the output.
Silk Server is implemented in Scala7 and runs as a Servlet on the Jetty Web Server8. The REST interface has been realized using the Lift Web Framework9. The Silk Link Discovery Framework including Silk Server can be downloaded from the project homepage10 under the terms of the Apache Software License. 4
This section reports on the results of an experiment in which we used Silk Server to generate RDF links between authors and publications from a Semantic Web Dog Food Corpus dump and a stream of FOAF pro les that we crawled from the Web. Semantic Web Dog Food Corpus publishes information on people and publications from Semantic Web conferences. FOAF is a widely used vocabulary to describe persons, their connections, projects, publications and interests. Twitter is a social networking and microblogging website which provides user information as RDFa. Given these di erent sources for information on persons, the experiment aims at linking duplicate person descriptions. In the following, we explain the Silk-LSL speci cation used by Silk Server in the experiment; we then rst describe the setup of the experiment and nally report on and discuss the results of the experiment.
4.1
Figure 3 contains the link con guration used in the experiment for linking data items describing the same person. The complete link con guration for discovering RDF links between persons as well as publications is available online11.
The involved data sources for this experiment are the Semantic Web Dog Food Corpus dump (line 5) and an RDF input stream (line 9).
A link con guration may contain several link speci cations if links for different types of data items should be generated. Silk Server will set owl:sameAs links between duplicates as con gured in line 16.
Link speci cations contain link conditions which de ne the conditions that data entities must ful ll in order to be interlinked. Link conditions may apply similarity metrics to multiple property values of an entity or related entities. The resulting similarity scores can be combined and weighted using various similarity aggregation functions.
Link Conditions The link condition speci es how two data entities are compared for similarity. It consists of a number of comparison operators which are combined using aggregation functions.
A comparison operator evaluates two inputs and computes their similarity based on a user-de ned metric. Silk provides several similarity metrics including string, numeric, date, and URI similarity. String comparison methods cover the most common ones like Jaro, Jaro-Winkler and Levenshtein. Silk can easily be enhanced with new metrics.
Multiple comparisons can be aggregated using a speci c aggregation method by using the <Aggregate> directive.
In the given experiment's link condition we compute similarity values for the FOAF names, homepages, and mailbox hash sums (lines 24 to 45). The overall similarity value of two data entities is derived by the weighted average of the similarity values of all comparisons. To identify a person uniquely, either a homepage or a mailbox hash sum is required. Thus, two persons are considered equal if both names and either the homepage or the mailbox hash sum match.
Some comparison operators might be more relevant for the correct establishment of a link between two resources than others and can therefore be weighted higher. If no weight is supplied, a default weight of 1 will be assumed. As a person may be known under di erent names, matching homepages or mailbox hash sums are more important and therefore weighted higher (line 35). Filtering The generated links can be ltered by using the <Filter> directive. A threshold for the minimum similarity of two data items required to generate a link between them can be de ned (line 47). The number of links originating from a single data item can be limited. Only the highest-rated links per source data item will remain after the ltering. 11 http://www4.wiwiss.fu-berlin.de/bizer/silk/linkspecs/persons_and_ publications.xml persons from which 2,580 provide either a homepage or a mailbox hash which is required to uniquely identify them. We have set up a Linked Data crawler which takes a number of FOAF pro le URIs as seeds and follows linked pro les. The crawled documents are forwarded to Silk Server which generates owl:sameAs links to known persons from the Semantic Web Dog Food Corpus. All generated links have been written to an ouput le which has been analyzed for the results presented in section 4.3.
The crawler was also used to traverse the RDFa of Twitter accounts for which the server identi ed the corresponding persons in the Semantic Web Dog Food Corpus if any.
In order to show the exibility of Silk Server, the link con guration was further enhanced to also match publications. For this purpose the crawler was employed to also follow publication links in addition to FOAF pro les. 4.3
Generated links to FOAF pro les At rst, we evaluated how exhaustive the found links are. For this purpose, we exploited the fact that for 56 persons the Semantic Web Dog Food Corpus already sets links to their FOAF pro le. For 51 of these persons, Silk Server was able to reconstruct links from the stream. For some persons even multiple duplicated pro les could be identi ed. For example e.g. in addition to Tom Heath's12 o cial FOAF pro le <http://tomheath.com/ id/me>, Silk Server also identi ed him on <http://www.eswc2006.org/people/ #tom-heath>. Because in some cases, Silk Server found a link to another pro le than the one given in the data set, we checked all links manually for correctness. Thereby, all generated links have been found to be correct.
Next, we evaluated for how many persons in the Semantic Web Dog Food Corpus, the server was able to generate links to a FOAF pro le. In total, Silk Server was able to nd pro les for 228 persons in the data set. Thus, Silk Server was able to discover links to the FOAF pro le of additional 177 persons for which the Semantic Web Dog Food Corpus did not contain a link yet.
Dog Food Corpus, Silk Server was able to nd a corresponding Twitter account. Silk Server was able to detect more than one account for persons holding multiple accounts. For example, it found that Ralph Hodgson13 not only uses the account http://twitter.com/ralphtq but also the account http: //twitter.com/oegovnews.
Generated links to publications For 37 publications in the Semantic Web Dog Food Corpus Silk Server was able to nd the corresponding publication in the Web of Data. The number of links is lower than the number of found FOAF pro les because many persons do not link their publications in their pro le.. One exception is the Digital Enterprise Research Institute (DERI), which publishes the meta data about all publications as RDF14. 12 http://data.semanticweb.org/person/tom-heath 13 http://data.semanticweb.org/person/ralph-hodgson 14 http://www.deri.ie/publications/
Discovering links between data items across data sets requires record linkage and
duplicate detection techniques. There is a large body of related work on these
topics within the database community [
Silk builds on the research results from within these communities. Silk can be used in scenarios where di erent types of links should be discovered between Web data sources which often make use of terms from di erent vocabularies.
Besides Silk, there are two related tools for generating RDF links:
LinQuer [
Related work that also focuses on Linked Data includes Raimond et al. [
While LinQuer and GNAT only allow batch processing, Silk Server is the rst identity resolution component that works on an on-demand fashion and can be used together with RDF data streams.
The EU-funded project OKKAM17 o ers an Entity Name System (ENS), which supportes the storage and reuse of global entity identi ers. While OKKAM ENS contains several matching modules per default, it does not provide a exible and comprehensive link speci cation language.
The RKBExplore sameAs service18 is targeted at providing a uni ed view over multiple data sources by managing owl:sameAs links to identify duplicate URIs. In contrast to Silk Server the links are not generated based on user-de ned link speci cations, but must be provided to the system from external sources. 6
Vint Cerf, the inventor of the internet, said in his keynote speech at 19th International World Wide Web Conference (WWW2010) that in the age of the internet where everything should be connected, he would also expect database management systems to automatically connect new records that are added to a database with all related entities that are already stored in the database. With 15 http://www4.wiwiss.fu-berlin.de/bizer/d2r-server/ 16 http://virtuoso.openlinksw.com/whitepapers/relational%20rdf%20views%20mapping.html 17 http://www.okkam.org/ 18 http://www.rkbexplorer.com/sameAs/ Silk Server, we make a rst step to provide such functionality for the Linked Data context. 7
This work was supported in part by Vulcan Inc. as part of its Project Halo (www.projecthalo.com) and by the EU FP7 project LOD2 - Creating Knowledge out of Interlinked Data (http://lod2.eu/, Ref. No. 257943).