In the last years, there has been an increase in the amount of real-time data generated. Sensors attached to things are transforming how we interact with our environment. Extracting meaningful information from these streams of data is essential for some application areas and requires processing systems that scale to varying conditions in data sources, complex queries, and system failures. This paper describes ongoing research on the development of a scalable RDF streaming engine.
Extracting information from data streams is complex because of the heterogeneity of the data, the rate of data generation, the high volumes, and the often unclear data provenance. One use case would be real-time monitoring of public transportation in a city. Here, decisions on unexpected events, such as a car crash, should be taken on short time slots based on a set of spatio-temporal data streams coming from di erent providers. For instance, by diverting a bus line route. This means reasoning over data in the temporal order it is ingested by the processing system. Linked Stream Data may help to integrate datasets from di erent providers and to solve interoperability problems. Yet, remaining challenges require solutions from a stream processing approach. In the remainder of the paper we discuss some of these challenges (section 2), a general description of our approach towards e cient processing of RDF streams (section 3), and open questions to debate during the workshop (section 4). 2
The development of scalable services for real-time stream processing involves support for high throughput, management of complex queries, low latency response, fault-tolerance, and statistics extraction, among others. Nowadays, cloud services o er solutions to some of these problems, e.g. by applying elastic load balancing in presence of input data bursts. We mainly focus on the e cient processing of user queries over RDF streams (C.1), which requires parallelization at the query operator level.
Distributed computing refers to the processing of data in distributed systems. The continuous transmission of data (C.2) between sources and processing nodes, and among nodes may cause response delays that should be minimized when possible.
The integration of historical and real-time data with background knowledge (C.3) is challenging in Web-scale environments. Many RSP systems combine background knowledge with real-time processing, but historical data management is often overlooked [15]. Traditional systems tend to store data aggregates in order to save space, but with the capacity of current systems it is possible (and recommended) to store all data in raw format and de ne views on data batches [16]. The e cient management of historical data is essential to detect trends in data, extract statistics, or compare old data with current data to identify anomalies [15]. For instance, by aggregating metro users on the last hours and compare the numbers to the average of users during the last days. 3
E
cient Processing of Queries over RDF Streams This section gives an high level overview of our approach, which is motivated by the challenges described above. We address three aspects of stream processing: adaptivity in query processing, data compression, and the architecture choice for integrating real-time and historical data. 3.1
Heterogeneous data streams are generated from di erent sources, at di erent rates, and include multiple domains. Our purpose is to build a distributed stream processing engine capable of adapting to changing conditions while serving complex continuous queries. Some sources already generate Linked Data streams [3]. Otherwise, we provide a layer serving an ontology-based access to non RDF data stream sources. Adapters for various input formats, such as CSV or REST APIs, are used to convert heterogeneous streams to RDF.
To reach an e cient query processing over data streams (section 2, C.1) we will focus on query execution planning. Traditional databases include a query optimizer that designs an execution plan based on the registered query and data statistics. In a distributed stream processing environment, there are several aspects to contemplate: changing rates of the input data, failure of processing nodes, and distribution of workload, among others. Adaptive Query Processing (AQP) techniques [7] allow adjusting the query execution plan to varying conditions of the data input, the incoming queries, and the system. Additionally, it is used to correct query optimizer mistakes and cope with unknown statistics [2].
First, we will analyze di erent strategies to process query operators, such as JOIN or FILTER. There are various examples in the literature that use ordering approaches for a more scalable stream processing, e.g. in the implementation of JOIN operators [12] or in eviction strategies [11]. We will design Storm8 topologies to e ciently process a set of common operators based on parallelizable tasks. Storm topologies are formed by spouts and bolts. Spouts are stream sources, whereas bolts are stream processors. Trident is a high-level abstraction on top of Storm that allows for stateful stream processing.9 For instance, a windowed join can be implemented using stream snapshots, which in Trident are called states. Depending on criteria such as the selectivity of the join or the size of snapshots, the processing engine can decide on the more appropriate join operator in terms of e ciency. Then, we will de ne a list of queries for a speci c use case and will extend the topologies to t the queries. While new data is entering the system, a dedicated bolt will manage stream data statistics to reassign a di erent topology if data stream conditions vary. In Storm, the coordination between the master node, which controls the assignation of tasks to spouts and bolts, and the worker nodes is managed by Zookeeper.10 However, Zookeeper does not provide elastic load balancing o -the-shelf. The use of cloud services for this purpose, such as the one o ered by Amazon EC2, will be addressed in the near future. 3.2
To date, universal compressors (e.g. gzip) and speci c RDF compressors (e.g. HDT [9]) are commonly used to reduce RDF exchange costs and delays on the network. These approaches, though, consider a static view of RDF datasets, 8 http://storm.incubator.apache.org/ 9 https://storm.incubator.apache.org/documentation/Trident-tutorial.html 10 http://zookeeper.apache.org/ disregarding the dynamic nature of RDF stream management. A recent work [10] points out the importance of e cient RDF stream compression and proposes an initial solution leveraging the inherent data redundancy of the RDF data streams. Based on this proposal, we are currently working on a compressed data structure speci cally designed for the particularities of dynamic RDF streams [8]. In particular, we aim at providing a lightweight serialization of RDF streams which i) minimizes the data exchange among processing nodes (section 2, C.2) while ii) serving a small set of operators on the compressed data. Our engine will address real-time processing on the Web of Things context following the Lambda principles [16]. Lambda is a 3-layer architecture designed to t the requirements of Big Data: a batch layer (using Hadoop11) stores all the incoming data in an immutable master dataset and pre-computes batch views on 11 http://hadoop.apache.org/ historic data (section 2, C.3); a serving layer (NoSQL database) indexes views on the master dataset; and a speed layer manages the real-time processing issues and requests data views depending on incoming queries. The cloud platform to deploy our engine should provide computing services that scale on demand, as well as elastic load balancing. 4
Next steps on the short term will address the implementation of an scalable RDF stream processing system. Open questions that we would like to discuss at the workshop are: { How does the order of tuple arrival a ect the parallelization of join processing tasks? { Are the spatial (or spatio-temporal) properties of a tuple a dimension to have into account for ordering? In this case, what in uence does it have on reasoning tasks? And on parallelization tasks? { How does the out-of-order tuples a ect the processing of streams? In case of discarding them, how to communicate this decision in the results?
The research leading to these results has received funding from the European
Union's Seventh Framework Programme (FP7/2007-2013) under grant
agreement no. 257641, PlanetData network of excellence, and from Ministerio de
Economa y Competitividad (Spain) under the project "4V: Volumen, Velocidad,
Variedad y Validez en la Gestin Innovadora de Datos" (TIN2013-46238-C4-2-R).
5. Calbimonte, J.P., Jeung, H., Corcho, O., Aberer, K.: Enabling Query
Technologies for the Semantic Sensor Web. International Journal on Semantic Web and
Information Systems 8(
Trends in Databases 1(
Springer Berlin Heidelberg (2013)
14. Le-phuoc, D., Parreira, J.X., Hauswirth, M.: Linked Stream Data Processing. In:
Reasoning Web. Semantic Technologies for Advanced Query Answering, pp. 245{
289. Springer Berlin Heidelberg (2012)
15. Margara, A., Urbani, J., van Harmelen, F., Bal, H.: Streaming the web: Reasoning
over dynamic data. Web Semantics: Science, Services and Agents on the World
Wide Web 0(0) (0)
16. Marz, N., Warren, J.: Big Data: principles and best practices of scalable realtime
data systems. Manning Publications (2013)
17. Rinne, M., Nuutila, E., Seppo, T.: INSTANS: High-Performance Event Processing
with Standard RDF and SPARQL. In: Henson, C., Taylor, K., Corcho, O. (eds.)
Proceedings of the 5th International Workshop on Semantic Sensor Networks. pp.
81{96. CEUR-WS, Boston, Massachusetts, USA (2012)
18. Sequeda, J.F., Corcho, O.: Linked Stream Data: A Position Paper. In: Proceedings
of the 2nd International Workshop on Semantic Sensor Networks, SSN 09.
CEURWS, Washington, USA (2009)
19. Sheth, A., Henson, C., Sahoo, S.S.: Semantic Sensor Web. IEEE Internet
Computing 12(