Despite the growth of the number of LOD datasets and the increasing variety of covered topical domains, most of the links connecting RDF resources of different datasets are identity links (often between descriptions that match perfectly) and a high number of datasets still do not contain out-links. The creation of different types of links and the process of analyzing the Linked Data space for new interlinking possibilities is time-consuming and tedious. This paper describes a crowd-powered approach to knowledge integration, which aims at supporting data publishers in designing new interlinking processes, as well as validating and enhancing automatically computed links.
Data interlinking is one of the critical tasks1 towards the realization of the global data
space on the Web [
Methods that address these issues relying exclusively on machine computation [
This paper presents CROWDKI, a system that automatically collects human input that becomes useful for two steps of data interlinking: (1) the design of interlinking processes and (2) the enhancement of automatically computed links. In order to do so, CROWDKI uses microtask crowdsourcing. 2
CROWDKI2 is a system that automatically creates and publishes microtasks (i. e. simple
tasks) in online labor marketplaces (e. g. Clickworker, ClixSense, etc.), in which people
all around the world and with different backgrounds [
Assessing the relevance of different interlinking possibilities: given a list of interlinking possibilities (i. e. D1.uriClass1,uriPredicate,D2.uriClass2) applicable to pairs of RDF datasets, the DCAT and VoiD descriptions of the content of the datasets3 and a description of the context in which the integrated data will be consumed, CROWDKI generates survey-style microtasks to ask humans how relevant the information enabled by the interlinking possibilities is for the specific context. Figure 1a shows the kind of questions that CROWDKI asks the crowd to collect relevance judgments. The context in this example is the official Web site of a car company and there are three interlinking possibilities: connecting Cars and Persons with any of the predicates wasDesignedBy, wasDrivenBy,wasRecommendedBy. For each of these possibilities CROWDKI asks (1) to rate the relevance of the type of information in the context of the official Web site of the company, (2) to (voluntarily) explain the reason of such judgment, and (3) to classify the relevance judgment. The last question is intended to get further information about the reason that the type of link less relevant (i. e. the predicate or the objects of the target dataset).
Validating and Enhancing automatically computed links: given two RDF datasets and a set of candidate links (e. g. from a link discovery algorithm), CROWDKI generates a set of microtasks to ask the crowd to review each candidate link. The set of candidate links can include links accepted and rejected by an automatic interlinking tool in order to be able to validate and enhance results. CROWDKI can also generate the Cartesian product set and a balanced set of correct and incorrect links based on a reference interlinking (however, these link generators do not currently support large datasets). Figure 1b shows an example of such microtasks, in which the description of two RDF resources is displayed and the user is asked about the relation between the two resources. 2.2
CROWDKI uses CrowdFlower4 to publish microtasks, because it distributes microtasks in multiple marketplaces reaching millions of users, and it provides support for gold standardbased quality assurance (i. e. microtasks with a known answer, used for instructing and testing the accuracy of crowd workers). The communication between CROWDKI and CrowdFlower is done using the CrowdFlower RESTful API5. CROWDKI is implemented in Java and it is divided into different components grouped in packages according to the microtask management cycle: Microtask generation This package includes functionality for parsing and preparing the data to be included in the microtasks (either interlinking possibilities or candidate links), as well as classes for generating the different types of microtasks (i. e. different templates for each of the use cases). Microtasks in CrowdFlower are created using
the CrowdFlower Markup Language6, in combination with HTML markup, and Javascript (if needed). The basic structure of the CROWDKI microtask templates are stored in separate text files, therefore, they are reusable and extensible. The microtask settings (e. g. the number of trusted judgments required per microtask, the number of microtasks per page, the payment, number of minimum gold microtasks required to pass) are read from the configuration file. Apache Jena7 is used to parse and query RDF data with SPARQL (to get the list of property values to be included in the microtasks—also defined in the configuration file). The RW access to other non-RDF files is done using the Guava IO library8.
Microtask publication This package contains the classes for publishing the generated microtasks in CrowdFlower. Gold microtasks are created separately from the normal microtasks, but following the same design. The data for such microtasks are provided directly from an RDF file. While it is possible to create the gold microtasks programmatically, it is better to write explanations on the gold microtasks manually, because even if this requires some time, gold answers need to explain crowd workers the way the specific microtasks work. CrowdFlower offers the possibility to launch microtasks including a QuizMode (i. e. crowd workers have to pass a test with only gold microtasks to be able to work on the rest of the microtasks). Targeted crowdsourcing is out of the scope of CrowdFlower, but it is possible to select between high speed or high quality workers, filter geographical regions and restrict the work to people speaking a particular language. CROWDKI may be easily extended to support other crowdsourcing platforms.
Response collection Once the required number of trusted workers has accomplished the microtasks, CrowdFlower generates several reports containing the results and information about the crowd workers (e. g. the marketplace they have worked from, their location and the time spent on the microtask). The platform also generates further statistics which can be seen exclusively from the requester GUI (e. g. the agreement of crowd workers with the majority vote). Such reports are the output of the first CROWDKI use case (i. e. assessing the relevance of interlinking possibilities). However, for the realization of the second use case (i. e. enhancement of generated links), CROWDKI contains further classes to process the responses of the crowd, collect the aggregated response (defined by majority voting) and serialize the crowd interlinking in N-Triples. 3
Communication is key for success: crowd workers require detailed instructions and contextual information about the data. Crowd workers complained about gold standard questions that indicated that two persons were the same when they had a different date of birth—without knowing about the existence of typos in the data, their claim made perfect sense. Iterating after collecting feedback from crowd workers (also through forms outside the platform) can improve the task design considerably.
Twitter and specialized forums, and they report on their satisfaction with accomplished/offered work. Adopting de facto standards (e. g. the reward to pay) it is important to be competitive with regard to other requesters.
at a cost, it is better to restrict its use to cases that require it—connecting datasets by location when both datasets contain country ISO codes can be perfectly done by Silk9. Testing the feasibility of a subset of the data can save money and time. The limitations are that CROWDKI requires several hours/days to get input from the crowd, and that selecting particular crowd workers in CrowdFlower may only be done after testing the people and programming workarounds (e. g. filtering worker IDs with Javascript / publishing multiple sets of microtasks). 4
The use of human computation in Semantic Web tasks has been acknowledged as an
effective way to overcome the limitations of automatic methods [
The system introduced in this paper enables crowd-powered knowledge integration on
the Web of Data. Dataset recommendation systems for interlinking could leverage the
human labels that CROWDKI can collect about the different interlinking possibilities,
and combine this information with other automatically extracted criteria such as the
current popularity of LOD datasets as defined by DING![
10 http://goo.gl/6hfuTk 11 Linked Open Vocabularies http://lov.okfn.org/dataset/lov/
The author would like to thank Elena Simperl, Steffen Staab, Natasha Noy and Matthias Thimm for the discussions about crowdsourced interlinking. The research leading to these results has received funding from the European Union’s FP7 under grant agreement no.611242-Sense4Us project.