We demonstrate RDFpro (RDF Processor), an extensible, generalpurpose, open source tool for processing large RDF datasets on a commodity machine leveraging streaming and sorting techniques. RDFpro provides out-of-thebox implementations - called processors - of common tasks such as data filtering, rule-based inference, smushing, and statistics extraction, as well as easy ways to add new processors and arbitrarily compose processors in complex pipelines.
1 Introduction
Processing of RDF data – e.g., Linked Open Data (LOD) – often requires performing
a number of common processing tasks such as triple-level filtering and/or
transformation, inference materialization, owl:sameAs smushing (i.e., replacing URI aliases with
a “canonical” URI), and statistics extraction. Although tools do exist for these tasks,
a Semantic Web practitioner typically faces two challenges. First, tool support is
fragmented, often forcing a user to integrate many heterogeneous tools even for simple
processing workflows. Second, tools coping with LOD dataset sizes in the range of
millions to billions of triples often require distributed infrastructures such as Hadoop
(e.g., WebPIE [
Given these premises we demonstrate RDFpro (RDF Processor) [
RDF ... LOD effects
(a) resulting pipeline processor ...
(b)
RDF Processor A processor @P (Figure 1a) is a software component that consumes
RDF quads from an input stream in one or more passes, produces an output stream of
quads, and may have an internal state as well as side effects like writing RDF data.
An RDF quad is a triple with an optional fourth named graph component, which is
unspecified for plain triples and triples in the default graph of the RDF dataset. Streaming
characterizes the way quads are processed: one at a time, with no possibility for the
processor to “go back” in the input stream and recover previously seen quads. Sorting
is offered to processors as a primitive to arbitrarily sort selected data during a pass. This
primitive is realized on top of the native sort Unix utility that support external sorting,
using dictionary encoding techniques to compactly encode frequently used RDF terms
(e.g., TBox ones) in order to reduce the size of sorted data and improve performances.
Sequential and Parallel Composition Composition can be applied recursively to build
pipeline processors starting from a fixed set of basic processors. In a sequential
composition (Figure 1b), two or more processors @Pi are chained so that the output stream
of @Pi becomes the input stream of @Pi+1. In a parallel composition (Figure 1c), the
input stream is sent concurrently to several processors @Pi, whose output streams are
merged into a resulting stream using one of several possible set operators, such as union
with/without duplicates. Composition supports complex processing tasks that cannot be
tackled with a single processor. Moreover, executing a pipeline processor is often faster
than executing the processors it is composed of separately (if separate execution is
possible), as input data is parsed once and I/O costs for intermediate files are eliminated.
Builtin Processors The basic processors included in RDFpro are listed below:
@read Reads RDF file(s), emitting their quads together with the input stream. Files
are read in parallel and, where possible, split in chunks that are parsed concurrently.
@write Writes quads to one RDF file or splits them to multiple files evenly; quads are
also propagated in output. Parallel, chunk-based writing is supported as for @read.
@download Emits data downloaded from a SPARQL endpoint using a query.
@upload Uploads input data to an RDF store using SPARQL INSERT DATA calls.
@tbox Filters the input stream by emitting only quads belonging to TBox axioms.
2 http://rdf4j.org/
@transform Discards or rewrites input quads one at a time, either based on simple
matching criteria or based on an arbitrarily complex JavaScript or Groovy3 script.
@smush Performs smushing, replacing the members of each owl:sameAs
equivalence class with a canonical URI selected based on a ranked namespace list.
@rdfs Computes the RDFS deductive closure of an input stream consisting only of
ABox quads. A fast, hard-coded implementation loads the TBox from a file and
computes its closure first, using the resulting domain, range, sub-class, and
subproperty axioms to perform inference on quads of the input stream one at a time.
@rules Emits the closure of input quads using a customizable set of rules. Rules
heads and bodies are SPARQL graph patterns, with FILTER, BIND, and UNION
constructs allowed in the body. The current implementation is based on Drools.4
@mapreduce Applies a custom map script (JavaScript or Groovy) to label and group
input quads into partitions, each one reduced with a reduce script. A multi-threaded,
non-distributed MapReduce implementation based on the sort primitive is used.
@stats Computes VoID [
Using RDFpro RDFpro binaries and public domain sources are available on its website.5 RDFpro can be used in three ways: (i) as a command line tool able to process large datasets; (ii) as a web tool suited to smaller amounts of data uploaded/downloaded with the browser; and (iii) as a Java library6 embedded in applications. Users can extend RDFpro via custom scripts and rulesets, while developers can create new processors by implementing a simple Java API and focusing on the specific task at hand, as efficient streaming, sorting, I/O, thread management, scripting, and composition facilities are already provided.
Examples of using RDFpro as a command line and web tool are shown in Figures 2a
and 2b, where a pipeline is executed to compute the RDFS closure of some DBpedia
data (70M triples) and return only rdfs:label triples of entities of type dbo:Company.
The pipeline performs 6 tasks: (i) read data; (ii) compute RDFS closure using DBpedia
TBox; (iii) keep rdf:type and rdfs:label quads; (iv) partition quads by subject, keeping
partitions with object dbo:Company; (v) retain rdfs:label quads; (vi) write results.
Examples of applications using RDFpro as an embedded Java library for RDF I/O, filtering,
smushing, RDFS and rule-based inference are the KnowledgeStore [
A video showing the usage of RDFpro is available on the website, together with a
fully-working installation of the RDFpro web interface, where users can try arbitrary
commands and processing tasks. The demo will mainly focus on using this web
interface on suitable examples, to demonstrate the usability of RDFpro in tasks such as the
ones considered in [