Due to the growing need to timely process and derive valuable information and knowledge from data produced in the Semantic Web, RDF stream processing (RSP) has emerged as an important research domain. In this paper, we describe the design of an RSP engine that is built upon state of the art Big Data frameworks, namely Apache Kafka and Apache Spark. Together, they support the implementation of a production-ready RSP engine that guarantees scalability, fault-tolerance, high availability, low latency and high throughput. Moreover, we highlight that the Spark framework considerably eases the implementation of complex applications requiring libraries as diverse as machine learning, graph processing, query processing and stream processing.
The Resource Description Framework (RDF) is a exible data format that was originally designed for the Web, and is now gaining popularity in the Internet of Things (IoT). A majority of IoT data are dynamically generated from various sources, e.g., sensors, and require inference services to meet their full analytic potential. This trend leads to the notion of RDF Stream Processing (RSP) which gains more and more attention as a research topic.
Due to the baseline de ned by some RSP benchmarks such as LSBench [
To meet the expectations of Big Data projects, a novel RSP engine is required. Motivated by the WAVES project1, this work provides insights on the integration of Apache Kafka, a distributed messagging broker, and Apache Spark, a distributed computing framework. These two Big Data frameworks will ensure robustness, reliability and scalability properties.
Applications based on stream processing frequently require di erent libraries and tools, to (1) perform incremental and iterative tasks, e.g., query processing and machine learning, (2) process graph data models and (3) handle all the streaming machinery, e.g., continuous query, windowing operations. Currently, two distributed stream processing frameworks have been emphasized by their adoption in large industrial projects: Apache Spark and Apache Flink. SparkStreaming is based on micro-batch execution mechanism, and provides the subsecond delay. Flink is another popular massively parallel data processing engine which supports real-time data processing and CEP. Due to the enrichment and the maturity of the platform ecosystems, we choose Spark Streaming as the framework of our RSP engine. 2
Our motivating use case concerns the industrial application of water resource network management. It aims to provide real-time analytics over RDF data streams. The observations, also denoted as events, are dynamically generated from various sensors and are hence anchored in spatio-temporal analytics. The measures we are considering correspond to pressure, ow, chlorine, temperature and turbidity. Their real-time analysis permit to detect water network anomalies, such as water leaks, and can have important impacts at the economical and environmental levels. Nevertheless, our goal is to design a generic RSP engine that can easily adapt to use cases concerned with other domains. Intuitively, the goal is to seamlessly integrate novel ontologies, data streams and sets of queries within a highly distributed, reasoning-enabled, continuous query and complex event processing system. 3
where t is the event timestamp. Then, the obtained RDF event streams are continuously sent to the Kafka message broker. We use Kafka to manage incoming event streams. Typically, each pre-de ned Kafka topic is associated to some speci c RDF events (e.g., the event of ow observation or chlorine observation, etc.). Finally, Spark-Streaming concurrently receives, caches and deserializes incoming data streams.
We combine Spark SQL, MLib and GraphX with Spark Streaming libraries to form the computing core of our system. It mainly covers the RDF data analytic in two aspects: basic timely SPARQL query processing and data mining over RDF streams. Indeed, the canonical SPARQL query processing with external and contextual data becomes valuable, which may also support stream reasoning. Moreover, the system is also planned to meet the requirement of advanced data analytic, such as classi cation and anomaly detection. Besides, Spark provides a seamless connection among its available libraries, since they share the same data collection abstraction, i.e. RDD. In the following, we present the tasks that highlight the use of these libraries.
Spark Streaming is an extension of the core Spark API and enables near realtime stream processing. We use Spark Streaming as the basic stream processing layer. It receives input data from Kafka, divides and parallelizes the data into batches, which are next processed by Spark Core.
Spark SQL provides a high-level abstraction to support distributed relational
operations. The work in [
MLib is a native Spark library for machine learning. We use MLib to run the data analytics pipeline. One of the target scenario is anomaly detection. Once the event is received by Spark Streaming, we push events to the analytic layer which uses classi cation algorithms and decision tree. For instance, we compare measures for the same geographical sector on two di erent dates using a clustering approach such as k-means.
GraphX facilitates parallel graph processing on Spark. We mainly use GraphX to generate semantic-aware dictionaries for our knowledge bases. Intuitively, it enables to seamlessly compute connected components, for instance to compute the transitive closure of some hierarchies or triples involving the owl:sameAs property. This computation only requires to provide two RDDs, containing vertices and edges of the RDF graph, to a given function. The output can easily be processed within our Spark core program. 4
RDF stream processing is an emerging area that is still in its infancy and hence requires to conduct much more research. In this work, we present our envisioned RSP system which is based on Spark Streaming and Kafka. These mature ecosystems bring important properties expected in Big Data applications and have proved to ease the design of application requiring libraries as diverse as machine learning, graph computations, query processing and of course stream management. In future work, we plan to extend our query processing with a trade-o between two common reasoning approaches, namely materialization and query reformulation. 5
This work is funded by the Fonds Unique Interministeriel (FUI #17) through the WAVES project, available at http://waves-rsp.org.