Advances in Data Management methods have resulted in a wide array of storage solutions having varying query capabilities and supporting di erent data formats. Traditionally, heterogeneous data was transformed o -line into a unique format and migrated to a unique data management system, before being uniformly queried. However, with the increasing amount of heterogeneous data sources, many of which are dynamic, modern applications prefer accessing directly the original fresh data. Addressing this requirement, we designed and developed Squerall, a software framework that enables the querying of original large and heterogeneous data on-the- y without prior data transformation. Squerall is built from the ground up with extensibility in consideration, e.g., supporting more data sources. Here, we explain Squerall's extensibility aspect and demonstrate step-by-step how to add support for RDF data, a new extension to the previously supported range of data sources.
The term Data Lake [
The value of a Data Lake-accessing system lays in its ability to query as much data as possible. For this sake, Squerall was built from the ground up with extensibility in consideration, so to allow and facilitate supporting more data sources. As we recognize the burden of creating a wrapper for every needed data source, we resort to leveraging the wrappers that data source providers themselves o er for many state-of-the-art processing engines. For example, Squerall uses Apache Spark and Presto as underlying query engines, both of which bene t from a wide range of connectors accessing the most popular data sources.
In this demonstration, we complement the published5 work about Squerall [
Squerall is an implementation of the Semantic Data Lake concept, i.e.,
querying original large and heterogeneous data using established Semantic Web
techniques and technologies. It is built following the Ontology-Based Data Access
principles [
As we recognize the burden of creating wrappers for the variety of data sources, we chose not to reinvent the wheel and rely on the wrappers often o ered by the developers of the data sources themselves or by specialized experts. The way a connector is used is dependent on the query engine: { Spark: the connector's role is to load a speci c data entity into a DataFrame using Spark SQL API. Its usage is simple, it only requires providing access values to a prede ned list of options inside a simple connection template: s p a r k . read . f o r m a t ( s o u r c e T y p e ). o p t i o n s ( o p t i o n s ). load Where sourceType designates the data source type to access, and options is a simple key-value list storing e.g., username, password, host, cluster settings, etc. The template is similar in most data source types. There are dozens connectors6 already available for a multitude of data sources. { Presto: access options are stored in a plain text le in a key-value fashion.
Presto uses directly SQL interface to query heterogeneous data, e.g., SELECT cassandra.cdb.product C JOIN mongo.mdb.producer M ON C.producerID = M.ID, there is no direct interaction with the connectors. Presto internally and transparently uses the access options to load necessary data on querytime. Similarly, there are already several ready-to-use connectors for Presto7.
Hence, while Squerall supports by default MongoDB, Cassandra, Parquet, CSV and various JDBC sources, interested users can easily provide access to other data sources leveraging Spark and Presto connectors8. 5 At ISWC-Resources track. 6 https://spark-packages.org/ 7 https://prestosql.io/docs/current/connector.html 8 Tutorial: https://github.com/EIS-Bonn/Squerall/wiki/Extending-Squerall How to feed the Squerall with RDF and other data nuts?
Supporting a New Data Source: Case of RDF Data
In case no connector is found for a given data source type, we show in this section
the principles of supporting a new data source. The procedure concerns Spark
as query engine, where the connector's role is to generate a DataFrame from an
underlying data entity. Squerall did not previously have a wrapper for RDF data.
With the wealth of RDF data available today as part of the Linked Data and
Knowledge Graph movements, supporting RDF data is paramount. Contrary to
the previously supported data sources, RDF does not require a schema, neither
xed nor exible. As a result, lots of RDF data is generated without schema.
In this case, it is required to exhaustively extract the schema from the data
on-the- y during query execution. Also, as per the Data Lake requirements, it
is necessary not to apply any pre-processing, and to directly access the original
data. If an entity inside an RDF data is detected as relevant to (part of) a query, a
set of transformations are applied to atten the (subject,property,object ) triples
and extract the schema elements needed to generate the DataFrame(s). Full
procedure is shown in Figure 1 and is described as follows:
1. First, triples are loaded into Spark distributed dataset9 of the schema
(subject : String, property : String, object : String).
2. Using Spark transformations, we generate a new dataset. We map (s,p,o)
triples to pairs: (s,(p,o)), then group pairs by subject: (s,(p,o)+), then nd
class from p ( p=rdf:type) and map the pairs to new pairs: (class,(s,(p,o)+])),
then group them by class (class, (s, (p, o)+)+). Each class has one or more
instances identi ed by `s' and contains one or more (p, o) pairs.
3. The new dataset is partitioned into a set of class-based DataFrames, columns
of which are the properties and tuples are the objects. This corresponds to
the so-called property table partitioning [
This procedure generates a (typed) DataFrame that can join DataFrames
generated using other data connectors from other data sources. The procedure
is part of our previously published e ort: SeBiDa [
Where NTtoDF is the connector's instance, options are the access information including RDF le path and the speci c RDF class to load into the DataFrame. 9 Called RDD: Resilient Distributed Dataset, a distributed tabular data structure.
RDF Triples ((ss11,, p1a,, o1_At1)) (s1, p2, o2_t2)
… (s2 ,(s2, ap,n, B) on_t3) (s2, pn+1, on+1_t4) ...
RDF Connector Triples Dataset
S: str P: str s1 p1 s1 p2
... s2 pn s2 pn+1 ...
Type A DataFrame (Relevant)
ID: Str p1: t1 p2: t2 ...
s1 o1 o2 ...
...
Type B DataFrame (Irrelevant)
ID: Str pn: t3 pn+1: t4 ...
s2 on on+1 ...
...
pm: str
om pn+r: str on+r
DataFrame
Joins
DS1 DS2
In this demonstration article10, we have described with more depth the extensibility aspect of Squerall in supporting more data sources. We have demonstrated extensibility principles by adding a support for RDF data. In the common absence of schema, RDF triples have to be exhaustively parsed and reformatted into a tabular representation on query-time, which only then can be queried. In the future, in order to alleviate the reformatting cost and, thus, accelerate query processing time, we intend to implement a light-weight caching technique, which can save the results of the attening phase across di erent queries. Beyond Squerall context, we will investigate making the newly created connector (currently supporting NTriples RDF) available in Spark Packages (connectors) hub for the public to be able to process large RDF data using Apache Spark. 10 Screencasts are publicly available from: https://git.io/fjyOO