The benchmarking of window-based RDF Stream Processing (RSP) engines has recently attracted the attention of the Stream Reasoning community. Solutions like LSBench, SRBench and CSRBench tried to fulfill the need of shared practices for RSP engine evaluations. However, an infrastructure for the systematic comparison of existing systems is still missing. In this paper, we propose the requirements and an architecture for Heaven Test Stand, a facility to foster Systematic Comparative Research Approach (SCRA) for window-based RSP engine. Heaven allows to design and systematically execute repeatable, reproducible and comparable experiments. As further contribution a working implementation of Heaven Test Stand is released as open source.
A Systematic Comparative Research Approach [
Social sciences, like sociology or economy, are extremely relevant fields [
Due to the multiple technology concepts involved, Stream Reasoning (SR) [
The relevant challenge for the entire community1 is now the RSP engine
evaluation. To the RSP benchmarking extent are currently available RDF Streams,
Ontologies and Continuous queries [
testing facility, but an infrastructure for systematically compare the RSP engine execution under controlled conditions is still missing.
In related research areas like databases, the results of the evaluation is
know given a schema of the data , the data, and a query. Therefore, there
is no need to study the systems dynamics at once. Unfortunately, this is not
valid for RSP engines where the execution semantic is different from system to
system [
In this paper we answer such research question presenting Heaven Test Stand an infrastructure to enable experiment design and execution for RSP engines.
Outline: Section 2 and 3 present respectively Heaven Test Stand requirements and architecture. Section 4 provides an example of experiment design an some details regarding an open source implementation of the Heaven Test Stand2. Finally, Section 5 comes to conclusion and presents the future works. 2
Our aim is studying the RSP engine dynamics, that requires on-line analysis of the engine behavior. SCRA fosters cross-case analysis and, thus, we need to guarantee the experimental conditions to be repeatable, reproducible, and comparable and also to describe the variables involved in the evaluation.
From the aerospace engineering we borrow the notion of engine test stand, a facility for to design and systematically execute experiments over engines. With a test stand, it will be possible to collect performance measurements during the engine execution and enable post-hoc evaluations. In the following we state the requirements for an RSP engine Test Stand, namely Heaven Test Stand (HTS).
Experiments Reproducibility requires HTS to guarantee: (i) engine compatibility (R.1), to allow the comparison of experiment results; (ii) data independence (R.2), to allow the system users to choose any relevant ontologies (R.2a) and to describe how the input data stream is defined (R.2b) (e.g. dataset/API, data model like RDF Stream etc.); (iii) query independence (R.3), to allow the system users to define and register a set of relevant queries from their domain of interest.
Compatibility with the RSP benchmarking state-of-the-art is crucial for HTS relevance, thus, Heaven design must be extensible at software level (R.4), i.e. theoretically each module can be replaced with one having the same interface, but different behavior, without affecting architecture stability. Due to (R.1), HTS must adopt an event-based architecture (R.5) as normally done by RSP engines and it must exploit a simple to parse RDF serialization for events (R.6) to minimize the cost of putting an existing RSP engine on the test stand.
A minimal measurements set for a relevant RSP engine evaluation is defined
in [
The RSP engines input-output relationship is non-trivial [
Finally, experiments Comparability requires HTS to support the collection of the performance measurements for post-hoc analysis (R.11). 3
1
2 Streamer
RSPEngine TestStand
Start
MB Experiment 1 Experiment 4 Heaven Response 2 Stimulus 3 Engine Response 3
Stop 4
MB 5
ResultCollector Data 5
Disk In this section we describe HTS architecture and also indicate which requirements are satisfied at architectural level.
The tuple < E , D, T , Q > describes the top-level input of HTS: an Experiment. E is the evaluated RSP engine (satisfying R.1); D is the description of the incoming data flow (satisfying R.2a); T is the ontology (satisfying R.2b); Q the continuous queries set registered into E (satisfying R.3).
In Step (1) HTS receives such an Experiment and it independently initializes each module: the engine E needs to be initialized by registering into the ontology T and all the queries in Q; the Streamer, which is the actual data stream source, uses the description D to build the incoming information flow (e.g. an RDF Streams) to push into E ; the Result Collector starts listening for the results of E , that it will persist for post-hoc analysis (satisfying R.11).
During experiment execution, HTS loops through the steps from (2) to (5) until the ending condition is reached (e.g the end of the dataset). It exchanges three kinds of events: (i) Stimulus (S) a portion of the input information flows in which all triples have the same timestamps; (ii) Engine Response (ER) the event format that E is required to output. It contains the answer to one of the query in the query-set Q registered in E given the ontology T and the active window in the RSP engine; (iii) Heaven Response (HR) which encapsulates the ER content adding the performance measurements collected by HTS.
In step (2), the Streamer builds and pushes to E an event S. Before starting step (3), HTS starts a timer to measure latency (satisfying R.4a) and it measures the memory load (satisfying R.4b).
In step (3), HTS invokes the engine E and then it waits until E completes its processing. We can be sure that HTS and the engine are concurrently executing, because HTS is a finite state machine. HTS stops the timer and measures again the memory load as soon as E returns it the control.
In step (4), HTS creates a HR adding to the produced ER the collected performance measurements. HTS pushes the HR to the Result Collector.
In step (5), the Result Collector persists HR content (satisfying R.11). 4
In this section we explain how to design each element of the tuple < E, D, T , Q > according to the current HTS implementation. We also motivate the development choices we did, by the means of the proposed requirements (Section 2).
The E specification is realized by the means of a facade pattern [8, pp. 243]. An abstract RSP engine class allows to associate any RSP engines ensuring engine independence (satisfying R.1), it intercepts the S and taking care of the ER creation.
To simplify the initial usage of HTS, the current implementation contains four naïve RSP engine implementations with external time-control, that we be natively used as E. Some results about their performances, evaluated using HTS current implementation are already available3.
D comprises all the details of the incoming input flows: (i) the actual data to stream (e.g. a dataset); (ii) the number of the incoming data streams, (iii) their data model (e.g. RDF Stream); (iv) their flow rate profile (e.g. Gaussian, Poisson) and an ending condition for the experiment (e.g. max number of events). The Streamer interface allows to describe all these details according with the user needs. The current HTS release comprises the RDF2RDFStream, that adapts any RDF dataset to a streaming scenario (RDF Stream) by adding to a Stimulus the required number of triples to build such an event. Different flow profiles can be realized by the means of the FlowRateProfiler. In our testing experiments, we adapted one generation of LUBM(1000, 0)4 to our purpose. 3 http://streamreasoning.org/TR/2015/Heaven/iswc2015-appendix.pdf 4 LUBM(N, S) indicates the dataset with N universities generated using a seed S. Moreover, RDF triples in S and E R are encoded in the N-Triple format5, an easy-to-parse RDF serialization used by the majority of exiting RSP engines (satisfying R.6).
The ontology T is chosen according with the background knowledge required for the reasoning. During the execution of an experiment, T is considered to be static. Indeed is a good practice to reduce the reasoning complexity by executing its materialization at setup-time. In the current implementation we used the RDFS version of the LUBM ontology as T .
The query set Q must be register directly to E , because HTS does not offer
API yet. For our evaluation, we used a single query for each engine for each
experiment. They were variants of the full the materialization under ρDF [
Notably, HTS is currently implemented as a single thread application (satisfying R.9). Completeness and Soundness (C&S) of the query results and the HTS ability to know exactly what was sent into E are evaluated post-hoc (R.7c) by using the content of the persisted HR events. Real time verification of C&S is also possible for those engines which allow external time control, because HTS satisfies (R.9), but it may violate requirement (R.10), due to the large memory footprint of reasoning procedures. 5
In this paper we presented the requirements and the architecture of Heaven Test Stand, an infrastructure to enable a Systematic Comparative Research Approach for RSP engines, by the means of experiment design and execution. A further contribution is a working implementation of HTS, already available on GitHub7.
Enabling SCRA is a crucial step towards an efficient and effective stream reasoning, but, as first future work, we have to prove HTS usability and effectiveness of the proposed implementation. Solving the former requires us to go step by step to the experiment design and execution; the latter, instead, requires to provide experimental proofs that HTS influences the engine performance in a controlled and predictable way only and it does not affect the query results.
Another future work consists into better comprehend the state-of-the-art
solution space, that means benchmarking mature solutions like the C-SPARQL [
We want to make available the performance analysis, together with the experiments setting we use for their evaluation (i.e. the < E , D, T , Q > configuration). This is clearly necessary, because we want HTS to be adopted by the entire SR 5 http://www.w3.org/2001/sw/RDFCore/ntriples/ 6 All the queries have the same sliding parameter β = 100 ms and they differ for the duration ω. In particular, we use time-based sliding windows in which ω is an integer multiple of the slide parameter β, i.e., it holds that ω = β ∗ S where S is a positive integer. 7 https://github.com/streamreasoning/heaven community. Thus, another pioneering step consists into developing a methodology for the result analysis. Such methodology should exploits statistical techniques to interpret the obtained results and allows us to finally answer questions like Qualitatively, which is the best solution? or Quantitatively, what distinguish a solution from other ones? or again Why solution A performs better than B under a certain experimental condition?
Acknowledgments. This work has been partially funded by the IBM faculty award 2013 granted to prof. Emanuele Della Valle and by the IBM PhD fellowship award granted to Daniele Dell’Aglio.