Due to the growing need to timely process and derive valuable information and knowledge from data produced in the Semantic Web, RDF stream processing (RSP) has emerged as an important research domain. Of course, modern RSP have to address the volume and velocity characteristics encountered in the Big Data era. This comes at the price of designing high throughput, low latency, fault tolerant, highly available and scalable engines. The cost of implementing such systems from scratch is very high and usually one prefers to program components on top of a framework that possesses these properties, e.g., Apache Hadoop or Apache Spark. The research conducting in this PhD adopts this approach and aims to create a production-ready RSP engine which will be based on domain standards, e.g., Apache Kafka and Spark Streaming. In a nutshell, the engine aims to i) address basic event modeling - to guarantee the completeness of input data in window operators, ii) process real-time RDF stream in a distributed manner - e cient RDF stream handling is required; iii) support and extend common continuous SPARQL syntax - easy-to-use, adapt to the industrial needs and iv) support reasoning services at both the data preparation and query processing levels.
Nowadays, the RDF data format is getting more and more popular in the Internet
of Things (IoT) ecosystem, i.e., data produced by sensors or other devices are
either directly represented as RDF triples or trasnformed into this standard. The
heterogeneous nature of IoT data sources potentially presents multiple challenges
for the implementation of real-time RDF Stream Processing (RSP) services.
These challenges refer to four distinct problematics [
To address the above-mentioned problems, we can identify two categories of
RSP engines which have designed in the last few years. The category of
centralized systems, including C-SPARQL [
Stream processing in general is one of the hottest topics in Big data. It is currently supporting analytics functionalities that have not been considered before. This is mainly due to the popularization of Web technologies (i.e., the pipeline that provides streams) as well as advancements in computing (i.e., emergence of distributed frameworks that facilitate the design of parallel computations).
RDF data is relevant in this streaming context because i) it is a popular graph format equipped with an expressive SPARQL query language, ii) it is anchored in the Web with its wide use of URIs and iii) it has the ability to support reasoning services.
The project within which my PhD research is taking place requires an important subset of the Semantic Web standards, namely RDF, RDFS and OWL considering facilities to represent and reason graph data and knowledge, SPARQL as a well-established query language and Linked Open Data to access external knowledge. The project concerns industrial water resources management. The events emitted from various sensors correspond to pressure, ow, chlorine, temperature, turbidity, etc. values and need to be processed almost in real-time to trigger system alerts. The analysis of these events is especially valuable if joined with external, contextual data such as the geographical properties where the sensor is situated.
Of course, our goal is to design a generic RSP engine that can adapt easily to use cases concerned with other domains. Intuitively, the goal is to seamlessly integrate novel ontologies, data/knowledge repositories and sets of queries within a highly distributed, reasoning-enabled, continuous query and complex event processing system. This aspect is particularly important since the PhD thesis is funded by a large IT company which envisions to have commercial activities using this system. 1 EP-SPARQL, a wrapper of ETALIS to support SPARQL-like syntax
In the last few years, a variety of RDF stream processing engines have been
proposed. As mentioned in section 1, the design of RSP needs to cover three
aspects [
Centralized RSP engines: currently, C-SPARQL, CQELS, SPARQLstream, and ETALIS/EP-SPARQL are popular centralized RSP engines. All of them are developed to run on a single machine.
Distributed RSP engines: CQELS-Cloud is the rst distributed RSP system
which mainly focuses on the engine elasticity and scalability. The whole system is
based on Storm2. Firstly, CQELS-Cloud compresses the incoming RDF streams
by dictionary encoding in order to reduce the data size and the communication
in the computing cluster. Then, to overcome the performance bottlenecks on join
tasks, the authors propose a parallel multiway join based on probing sequence,
which is inspired by the MJoin algorithm [
Katts is another RSP engine based on Storm, which applies graph partitioning 3 to optimize the message exchanging for cluster computing. Based on our evaluation, we can say that Katts allows a limited amount of query operators and queries with only-streaming data sources. Besides, neither query algebra optimization nor SPARQL syntax support have been considered. Finally, the implementation of Katts remains at the stage of scienti c prototype.
Distributed-ETALIS is a distributed Complex Event Processing (CEP) engine based on Storm and Kafka 4. Both Distributed-ETALIS and its centralized version ETALIS implement a rule-based DSL for event detection (pattern matching). It seems that ETALIS team has stopped the software maintenance. We also meet a serious scalability problem in our preliminary evaluation of ETALIS.
A distributed RSP system must rely on a generic distributed stream
processing framework such as Storm, Spark Streaming [
Given the context of this PhD thesis, the general research question is: How to e ciently query and reason over massive real-time RDF events data in a distributed computing environment? To answer this question, the following four aspects have to be considered: { Q1 - Distributed, robust RDF stream query processing: we consider the design of an architecture for the processing of RDF streams that is highly available, tolerates failures, scales and guarantees high throughput and low latencies. No RSP engine possesses all these properties and present performance gures comparable to non-RDF state of the art engines. In particular, these last engines are generally designed using a distributed pub/sub messaging system (e.g., Apache Kafka) and a dedicated stream engine (e.g., Apache Flink, Storm, Spark streaming, Beam). Replying to this question implies to adapt RDF peculiarities to this rapidly evolving ecosystem. { Q2 - Compression and reasoning: processing (i.e., querying and reasoning) in a real-time manner over complex graph-based events implies to optimize all computation aspects. We consider that a decompression-free querying approach tightly connected to a semantic-based ontology encoding can guarantee these properties. { Q3 - Extending continuous SPARQL query toward complex event processing capacities: we have identi ed some practical use cases that are requiring continuous query features that we have not encountered in available continuous SPARQL query languages, e.g., session-based windows. These laguage extensions have to be integrated in our declarative query language. { Q4 - How to evaluate RSP engines: we need a way to evaluate the system that is currently being designed. This implies to identify, experiment and evaluate streaming benchmarks and engines. A set of performance metrics also need to be de ned to evaluate distributed RSP engines in a cluster environment. 5
Some preliminary results have obtained for Q4 of section 4. The main idea was to evaluate three centralized RSP systems, i.e. C-SPARQL, CQELS, ETALIS. This evaluation de nes a complete set of performance metrics to measure query latency and memory consumption by varying stream rate, window size, number of streams , etc. The evaluation of distributed RSP engines are in progress, some
Fig. 1: Engine architecture performance metrics need to be added, such as measure the query latency and engine throughput by varying the number of machines. We have currently tested Katts on Amazon EC2 6 by using the real-world data. The evaluation results of Katts are kept for future comparisons with our system. This part of work gives us a deep understanding on existing RSP engines and continuous SPARQL query language features. This has helped me on the design and the implementation of our own RSP engine.
Recently, I started to develop a prototype 7 that will run continuous SPARQL queries on Spark Streaming and Kafka. In Figure 1, I provide a high-level view of the system's architecture. The data obtained from sensors or other sources are rst encoded as RDF events. Next, the obtained RDF event streams continue to be serialized into binary format and transmitted to Kafka. Then, SparkStreaming concurrently receives, caches and deserializes the incoming binary stream. Finally, the system applies the precompiled optimized logical plan to proceed the query execution. 6
In this section, I present how I intend to address the Q1, Q2 and Q3 research questions presented in section 4 as well as how I will validate the system e ciency.
Q1 addresses to three subquestions: i) Data/Event Modeling. Firstly, the
system aims to support two RDF formats for data updating: triple (i.e. (s; p; o)) by
triple and event by event. Basically, an RDF triple can be regarded as the
simplest RDF graph pattern. I use Sesame [
Due to the real-time aspect in streaming context, some optimization
techniques based on data preprocessing become hardly applicable, e.g. data indexing,
vertical partitioning [
Q2 concerns compression and reasoning aspects. I approach goes along the
work of ERI[
The continuous SPARQL query language we have implemented so far is inspired by C-SPARQL. It supports the de nition of di erent window forms, e.g., xed or sliding windows, via keywords which are permitted in queries. We have identi ed an additional session-based window form that is required in some practical use cases. Hence, the approach to address Q3 will consist in de ning the proper semantics for additional continuous SPARQL query clauses, implementing this semantics within our distributed streaming engine and integrating optimization for this extended query languages, e.g., to support e cient query reformulation. Finally, this novel approach will be heavily tested and evaluated against practical use cases and data sets. 7
Our system will be evaluated on both synthetic and real-world data sets. Concerning the synthetic data sets, we have already identi ed, experimented and even extended well-established benchmarks during our work of Q4. The realworld experimentation will be directly related to the water resources management use case. It consists of values obtained from real sensors. We aim to test the system live as well as replay old values and thus test whether our system is able to discovery peculiar situations that we known have occurred in the past.
The evaluation will be conducted both in local mode and distributed mode, with respect to the following outline.
Functionality support. Since our sytem aims to fully automatize the query execution, we need to implement all the common query operators ((BGP join, lter, aggregations, etc.). Correctness. On both synthetic and real-world data sets, we will be able to qualify the correctness of the provided answers. On our water resources management, we already know that correctness implies some reasoning. Performance. Two performance aspects need to be studied: Latency and throughput. Latency refers to the wall-clock time consumed by the engine on each query execution. Throughput depicts that how many RDF triples can be processed in a unit time. These two performance metrics gives us insights into whether the system could handle the target scenario or not. The engine throughput and query latency are recorded by varying the input stream rate (events/second) and the number of cluster nodes. 8
This PhD thesis is now at an early stage and preliminary results are promising. I still have a lot to design and implement but I'm getting closer to the point where room for thorough experimentation, evaluation, exploration and innovation will be possible. In the near future, I'm planning to propose some trade-o between materialization and query rewriting as well as novel optimization for RDF stream processing, e.g. querying over compressed data.