The ability to cross-link large scale data with each other and with structured Semantic Web data, and the ability to uniformly process Semantic Web and other data adds value to both the Semantic Web and to the Big Data community. This paper presents work in progress towards integrating Big Data infrastructures with Semantic Web technologies, allowing for the cross-linking and uniform retrieval of data stored in both Big Data infrastructures and Semantic Web data. The technical challenges involved in achieving this, pertain to both data and system interoperability: we need a way to make the semantics of Big Data explicit so that they can interlink and we need a way to make it transparent for the client applications to query federations of such heterogeneous systems. The paper presents an extension of the Semagrow federated SPARQL query processor that is able to seamlessly federated SPARQL endpoints, Cassandra databases, and Solr databases, and discusses future directions of this line of work.
Scalable, e cient, and robust data services are re-shaping the way that data analysis techniques are applied to the heterogeneous data cloud. Modern, distributed data storage and processing solutions o er unprecedented levels of scalability, robustness, fault tolerance and elasticity. They are, however, almost all outside the scope of Semantic Web technologies, as data values typically lack explicit semantics and cross-links.
The ability to cross-link large scale data with each other and with structured Semantic Web data, and the ability to uniformly process Semantic Web and other data adds value to both the Semantic Web and to the Big Data community; extending the scope of the former to include a vast data domain and increasing the opportunities for the latter to process data in novel ways and combinations. The technical challenges involved in achieving this, pertain to both data and system interoperability. We need a way to make the semantics of Big Data explicit so that they can interlink and we need a way to make it transparent for the client applications to query federations of such heterogeneous systems.
In this context, federated SPARQL query processing is a natural place to look
for a starting point in this e ort. Semagrow is such a federated query
processing system3 that provides a single SPARQL endpoint that federates multiple
remote SPARQL endpoints, transparently optimizing queries and dynamically
integrating heterogeneous data models by applying the appropriate vocabulary
transformations [
The opportunity to extend Semagrow beyond federations of triple stores in
a formally coherent way was presented by the results of the Comma Separated
Values on the Web (CSVW) working group of the W3C4 which has recently
nalized a suite of recommendations on the semantics of tabular data [
To make this more concrete, let us assume an example use case from the pilots
of the Big Data Europe project where our work is encouched [
In the remainder of this paper we will describe two alternative approaches to extend Semagrow so that it can construct federations of data sources other than SPARQL endpoints: using Semagrow-side connectors (Sections 2) or data source side SPARQL adapters (Section 3). We then discuss the relative merits of each and provide future research directions for this line of work (Section 4). 2
Cassandra o ers the scalability, fault-tolerance, and elasticity of modern distributed stores, but the Cassandra Query Language (CQL)5 is oriented towards processing tables and lacks the expressivity needed to join across tables. Moreover, Cassandra poses further restrictions on what queries that can be answered
even when expressible within CQL. For example, ltering predicates can only be applied to attributes declared as indexed by the schema of each database.
These access restrictions make the connectivity with Cassandra more challenging than, for example, in the case of a full enged SQL system. On one hand, not every SPARQL query can be directly translated into an equivalent CQL query. The query planner takes these restrictions into consideration and produce only valid plans that will directed towards a Cassandra source. On the other hand, Semagrow must compensate for the missing expressivity of Cassandra's queries by executing the remaining operations on its execution engine.
The connector is an open-source extension of the core Semagrow system.6 3
An alternative approach to the one described above is to provide the means of exposing non-RDF data as a SPARQL endpoint. The di erence is that the source presents itself as a SPARQL endpoint, rather than having a Semagrowside connector. To incorporate large amounts of textual data into a Semagrow federation, it is required to make data available as RDF and queryable using the OpenRDF Sesame API. Our approach to achieve this goal is to use Apache Solr as a backend for an OpenRDF Sail implementation. First, the use of Apache Solr gives us all advantages of a robust, scalable indexing framework, second the creation of an OpenRDF Sail Implementation based on Apache Solr will make such a full text index available for use with the SPARQL query language. It should be noted that Apache Solr is itself clusterable and can therefor be scaled and made highly available.
As a rst step, the foundation of the Apache Solr implementation has been layed out by creating a port of the OpenRDF Sesame API that is capable of creating Java 8 Streams of RDF Statements. Apache Solr's streaming capabilities can be translated to Java 8 Streams. Currently work is undertaken to implement an Apache Solr speci c version of the Semagrow Streaming Sail implementation. This will in fact combine the best of multiple worlds. It will be possible to use Lucene Search Syntax inside custom SPARQL functions, generally known as magic predicates . Additionally it will be possible to incorporate such an implementation seamlessly into the Semagrow federation. 4
The syntactic integration of diverse data sources can be implemented either by a Semagrow-side connector or by a SPARQL adapter in the data source side that translates SPARQL queries to valid queries of the wrapped data source. While the SPARQL adapter seems to be compatible with every SPARQL client it is not always trivial to develop one. The translation of an arbitrary SPARQL query to a single query of the target language might not be fully supported due
to limitation imposed by the underlying data source. In such cases the SPARQL adapter needs to plan and implement the missing functionality needed in order to support every SPARQL query.
On the other hand, a Semagrow-side connector is aware about the limitations of the source and the query planner will produce plans that are directly translated into the target query language. The query planner might also be presented with more optimization decisions since it considers the global execution plan of the query, and thus produce more e cient execution plans than using SPARQL adapters to achieve the same e ect. In other words, the Semagrowside connector will leverage the query planner of Semagrow to decide how to better access the underlying sources and the execution engine to compensate for the possible limitations. Based on the above the Semagrow-side approach seems more appropriate in situations where there is a signi cant expressivity gap between SPARQL and the target query language of the source whereas SPARQL adapters are preferable when not.
The current implementation of the aforementioned Big Data connectors rely on the fact that the underlying data storage supplies a central endpoint that interacts with external requests. This fact simpli es the connectivity between systems, but the central endpoint of data exchange may become a bottleneck in data intensive scenarios. Our immediate future plan is to more tightly integrate Semagrow with Cassandra, so that the Semagrow execution engine is itself distributed and exploits data locality in its interactions with the individual Cassandra nodes.
The work described here was carried out in the context of Big Data Europe: Empowering Communities with Data Technologies. Big Data Europe has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 644564. For more details, please visit https://www.big-data-europe.eu