ion makes our approach independent of the actual Hadoop version and thus ensures the compatibility to future changes of the Hadoop framework as they will be covered by the underlying Pig layer. We revisit PigSPARQL and demonstrate the performance improvement when simply switching the underlying version of Pig from 0.5.0 to 0.11.0 without any changes to PigSPARQL itself. Because of this sustainability, PigSPARQL is an attractive long-term baseline for comparing various MapReduce based SPARQL implementations which is also underpinned by its competitiveness with existing systems, e.g. HadoopRDF.
Today, MapReduce has been widely adopted in manifold application elds, especially in the broad area of Big Data, with Hadoop being the most prominent open source implementation. Though node e ciency is known to be rather poor, its success is mainly attributed to the inherent high degree of parallelism, robustness, reliability and excellent scalability properties while running on cheap and heterogeneous commodity hardware. Furthermore, new nodes can be added to the system on demand seamlessly at runtime.
Driven by the Semantic Web and Linked Open Data, new challenges with
regard to SPARQL query evaluation arise and scalability becomes an issue as
RDF datasets continuously grow in size, exceeding the capabilities of state of the
art non-distributed RDF triple stores [
In this paper we rst revisit PigSPARQL1, a mapping from SPARQL to the
query language of Pig [
PigSPARQL Architecture Pig is a data analysis platform on top of Hadoop with a fully nested data model, complemented by a comprehensive imperative query language (Pig Latin) that gives us a simple level of abstraction from the procedural model of MapReduce by providing relational style operators like lters and joins which are not available in MapReduce out of the box. We represent an RDF triple in the data model of Pig as a tuple of three atomic elds with schema (s, p, o). As we do not require any preprocessing and RDF triples are converted into the data model of Pig on the y, PigSPARQL is particularly suited for ad-hoc query processing, e.g. for ETL like scenarios where we do not want to build up a costly index structure in advance. In a typical SPARQL query the predicate of a triple pattern is usually bounded. Hence, PigSPARQL also supports optional vertical partitioning of the dataset as an additional preprocessing step.
Our mapping of SPARQL to Pig Latin follows a common design principle
based on an algebraic representation of SPARQL expressions (cf. Figure 1). First,
a SPARQL query is parsed to generate an abstract syntax tree which is then
translated into a SPARQL algebra tree. Next, we apply several optimizations on
the algebra level like the early execution of lters and a rearrangement of triple
patterns by selectivity. Finally, we traverse the optimized algebra tree bottom up
and generate for every SPARQL algebra operator an equivalent sequence of Pig
Latin expressions. At runtime, Pig automatically maps the resulting Pig Latin
script into a sequence of MapReduce iterations. More details are given in [
SPARQL Query Parser
Syntax Tree Algebra Compiler
Algebra Tree Algebra Optimizer
Algebra Tree Pig Latin Translator
Pig Latin Program Pig
MapReduce
SELECT * WHERE { ?person knows Peter . ?person age ?age
OPTIONAL { ?person mbox ?mb } FILTER (?age >= 18)} LeftJoin 4
BGP 3 ?person mbox ?mb
Filter 2 ?age >= 18
BGP 1 ?person knows Peter .
?person age ?age knows = LOAD 'rdf/knows' USING rdfLoader() AS (s,o); age = LOAD 'rdf/age' USING rdfLoader() AS (s,o); f1 = FILTER knows BY o == 'Peter'; t1 = FOREACH f1 GENERATE s AS person; t2 = FOREACH age GENERATE s AS person,o AS age; j1 = JOIN t1 BY person, t2 BY person; BGP1 = FOREACH j1 GENERATE t1::person AS person,
t2::age AS age; F = FILTER BGP1 BY age >= 18; mbox = LOAD 'rdf/mbox' USING rdf() AS (s,o); BGP2 = FOREACH mbox GENERATE s AS person,o AS mb; lj = JOIN F BY person LEFT OUTER, BGP2 BY person; LJ = FOREACH lj GENERATE F::person AS person,
F::age AS age, BGP2::mb AS mb;
STORE LJ INTO 'output' USING resultWriter();
1
Most of the published MapReduce based approaches are proof-of-concept
implementations, which are neither well documented nor running out of the box,
nor are available for public. Evaluation of such systems is time consuming and
easily leads to inexplicable results. This hampers the comparability of di erent
proposed solutions which is a key driver for further development. To introduce a
stable basis for comparison, we suggest PigSPARQL as an easy to use baseline
for SPARQL query processing with MapReduce because of the following reasons:
1. PigSPARQL is a reliable and stable system as it uses Pig as an
intermediate layer which is widely-used and maintained by Yahoo! Research. Pig's
processing framework is fairly competitive and continuously optimized and
enhanced with new features. This is con rmed by Figure 2.a that shows
exemplary the runtime improvement of PigSPARQL for SP2Bench Query 2
between Pig 0.5.0 and Pig 0.11.0 where we can observe a speed up by an order
of magnitude without changing a single line of code - other queries exhibit a
similar behavior as can be expected because of PigSPARQL's architecture.
2. For a comprehensive evaluation of di erent systems, they should be
installable and usable within a reasonable e ort, without the need of tricky con
gurations. In the context of such evaluations, PigSPARQL is very attractive.
The LUBM evaluation of HadoopRDF [
20 100 400 800 1200 1600 #triples (in million) PigSPARQL (Pig 0.5.0) PigSPARQL (Pig 0.11.0) 500 1000 1500 2000 2500 3000
#universities Merge Join (n-way) Merge Join (2-way) MAPSIN (n-way) MAPSIN (2-way) PigSPARQL (n-way) PigSPARQL (2-way) HadoopRDF
Conclusion. PigSPARQL is an easy to use and competitive baseline for the comparison of MapReduce based SPARQL processing. With the support of SPARQL 1.0, it already exceeds the functionalities of most existing research prototypes. For future work, we plan to add support for additional SPARQL 1.1 features.