Stream Reasoning (SR) is the research field that aims to solve the heterogeneity, noisiness, and incompleteness problems that come with the processing of continuously produced data. The use of Semantic Web technologies allows to mitigate many of these problems. RDF Stream Processing (RSP), a subfield of SR, specifically deals with the processing of RDF data streams. Many RSP engines and solutions exist, however, none provide the flexibility to rapidly create a prototype for new RSP applications or aid in the design of new components. In this demonstration paper, we showcase Streaming MASSIF Prototypes, a RSP prototyping platform that provides the necessary tooling to effortlessly create new RSP applications. Furthermore, the available tooling and the inherent flexibility of the platform allow researchers and developers to fully focus on the design of new RSP components, hiding superfluous complexities and configurations.
Data is being produced at a volume and velocity that out limits the ability to consume
the data as a whole [
Looking at most RSP engines and applications we can see that they can be decomposed into some abstract building blocks:
C-SPARQL [
CQELS [
CityPulse [
Streaming MASSIF [
Many more engines exist, most can be abstracted to the following building blocks: – Windowing: a function to divide the continuous stream into processable chunks. – Selection: a selection function a select and filter data according to a registered query. – Abstraction: a reasoning component to infer implicit and missing facts. – Temporal: a temporal reasoning component to infer temporal dependencies.
However, none of the existing engines or platforms provide any tools to easily load different data sources, handle the results, monitor the different building blocks, add some flexibility to reorder the building blocks, or add new components. We provide the needed tooling for easy prototyping new RSP applications and components. 3
To enable easy prototyping and integration of new RSP components, Streaming MASSIF Prototypes has been built on top of the following principles: 1. Flexible interfaces: in order to integrate new components, or compose building blocks in any order, interfacing between components should be flexible and transparent. Inspired on the flexibility of the Unix pipelines, we chose the use text as interface. This allows to easily plugin new components. Throughout the platform, we support semantic data using the Turtle serialization (or column separated tabular data if data has not been mapped to a semantic model yet). 2. Abstract components: all components are being abstracted on the highest level to PipeLineComponents, this allows that upstream or downstream components can be decoupled and are not depending on any specific implementation. To make prototyping easier, we support the following components: (a) Sources: the data ingestion points that load the data streams into the platform.
Support for Kafka, HTTP Post and get (with configurable timeouts), websockets and reading from file (with configurable timeouts). (b) Windows: function to window the data stream in processable chunks. Support for sliding and tumbling windows. Current implementation: Esper1. (c) Filters: allow to select and filter certain parts from the data stream through
SPARQL-queries. Current implementation: Apache Jena2.
(d) Abstractors: reasoning step that allows to infer implicit facts and abstract events
to high-level concepts. Current implementations: HermiT [
WebSockets, writing to file, and print sinks (to terminal or web interface). The benefit of these abstractions is that multiple implementations are possible. Note that additional components can easily be integrated. Figure 1 depicts an example pipeline configuration using the abstract components. 3. Monitoring: As depicted in Figure 1, each component is wrapped in a PipeLineComponent that has its own queue for asynchronous processing of the components, and a monitoring layer to allow easy evaluation of the component’s metrics. The metrics can be accessed through an HTTP endpoint and contain the processing times, the number of events in the queue, and the throughput. 4. Processing Graphs: Each component has the flexibility to have multiple inputs and outputs, allowing the creation of a processing graph for RSP applications. 5. Graphical Interface: To simplify the creation of the processing graph, we provide a drag and drop web interface that allows to compose RSP workflows. Figure 2 visualizes an example composition on the right pane. The left pane shows the registered query of the filter component and its monitoring details. Alternatively, the configuration can be loaded in JSON-LD format. 1 http://www.espertech.com/ 2 https://jena.apache.org/
In this demonstrator, we show how RSP applications, consisting of various components, can be built using the workflow composer. The demonstrator shows how complex applications can be built through drag and drop interaction and declarative definitions of the components. It shows how we can be debug RSP applications by investigating intermediate results of each of the components in the user interface and how each component can be monitored in order to detect bottlenecks. A video of the demonstrator can be found here3.
Acknowledgments: Pieter Bonte is funded by a postdoctoral fellowship of Fonds Wetenschappelijk Onderzoek Vlaanderen (FWO) (1266521N). 3 https://github.com/IBCNServices/StreamingMASSIF/wiki/Streaming-MASSIF-Prototypes