The relevance of many types of data perishes or degrades over time; to support timely decision-making, data integration systems must provide access to live data and should make it easy to incorporate new sources. We outline methods, based on web architecture that enable (near) real-time access to data sources in a variety of formats and access modalities. Our methods also enable rapid integration of new live sources by modeling them with respect to a domain ontology, and by using these models to generate a Linked Data interface to access them. Finally, we present initial experimental results of a scenario involving several static and dynamic sources from the web.
In recent years the number of web data sources and web APIs has increased
tremendously. The explosion is due to the overwhelming growth of available
Linked Data (e.g., [
However, integration of the di erent sources and APIs requires programmatic access over heterogeneous data sources that use di erent formats and protocols following di erent paradigms for accessing dynamic data. Further, the ability to quickly introduce new data sources into the application requires manual alignment and modeling, which is a labor-intensive and thus time-consuming process.
Integration of data sources in a variety of data source formats and access mechanisms is a di cult problem, especially as we want to add sources on-they, i.e., while the application is running. ? This research is based upon work supported in part by ONR Global under grant number N62909-13-1-N024, in part by the European Commission's Seventh Framework Programme FP7/2007-2013 under PlanetData grant number 257641, and in part by the National Science Foundation under grant number 1117913.
One challenge is to decide on the handling of static and dynamic sources. In many systems, dynamic sources push their updates in a streaming fashion to the target; the target systems handle static and dynamic data sources di erently. However, most sources on the web follow a request/response communication pattern based on polling of sources, with support for caching on the protocol level. We investigate the use of polling for frequently changing sources; the only difference between sources are their update rates. The update frequency of sources we currently consider is relatively low, ranging from seconds to months. One research question we address is whether a uniform interface to all data sources based on polling is a feasible approach in such a scenario. A main bene t of using a uniform interface is a simpli ed and more robust overall architecture.
Another challenge is the amount of manual e ort involved in building and maintaining an integration system. We want to reduce the time needed for including a new live source into a constellation of integrated sources. Thus, we aim for a exible approach based on declarative speci cations rather than procedural code. Declarative speci cations are used in the modeling of sources, and in the rules that encode how data sources are accessed and how data items relate to each other.
There are di erent strands of research targeting real-time data access. We
are exploring the trade-o outlined by [
The overall bene ts from the point of view of an application of such a method and apparatus are as follows: { We may achieve real-time access to data integrated from several sources, some of them static and some of them dynamic. { We can quickly integrate new data sources, as we use standard software interfaces to poll the current state of resources at speci ed time intervals or receiving updates and reacting to them, easing the transition from static to dynamic sources. { We can quickly integrate new data sources with the help of an application allowing for the semi-automatic modeling and lifting of non-RDF sources to the Linked Data level.
We begin with a scenario in Section 2, followed by an architectural overview in
Section 3. The main components, Karma [
The scenario includes a real-time visualization of the area that is augmented with current data from multiple sources to help a user make well-informed and timely judgments. Finally, we want to be able to integrate new sources rapidly; consider the addition of a new dynamic data source, for example the USC campus cruiser source which provides access to campus taxis via a JSON API.
Figure 1 shows the nal rendering of the integrated data in Google Earth3. Di erent sources have di erent update intervals; the bus sources, for example, are updated every several seconds, while others are updated once a month, following the manually determined update interval of the original API. A list of sources used in the prototype together with performance measurements can be found in Section 6.
We employ Linked Data as a uniform abstraction for sources. Sources following the Linked Data abstraction provide a standardized small set of supported operations and a uniform interface for both their data payload (RDF) and their fault handling (HTTP status codes). Such a common abstraction allowing the manipulation of states as primitives enables us to specify the interactions between components declaratively both on the operational and data levels. Consequently, we require mechanisms to quickly bring data sources to the Linked Data abstraction.
In our system we consider the following data artifacts: { D original data sources { O domain ontology { M Karma models based on domain ontology { L data sources as Linked Data { P data integration program, consisting of queries and rules
The main runtime components are Karma and the Data-Fu interpreter. Karma uses the data sources D in conjunction with the domain ontology O to create the models M; based on M, an execution component creates Linked Data interfaces L to the original data sources. The Data-Fu interpreter evaluates programs P that operate on L. We currently manually specify the update intervals for the evaluation of P, which ranges from seconds to months. Finally, Google Earth visualizes the query results. Figure 2 illustrates the components of our system.
Karma Modeling Services Sources
Design Time
Model Karma Linked Data Generator
Execution Time
To add a source (at design time), there are two places where manual e ort is involved: rst, a user has to model a new source in Karma; second, a user has to create a small Data-Fu program to specify what part of the data should be fetched via rules, together with a query that returns the nal results. Once the models and the rules are deployed and linked in the main KML le, the new sources are shown in the visualization component after the next refresh. 4
One of the challenges in our scenario is the rapid integration of new data sources, such as the service that provides status and location information for vehicles that transport students to and from the USC campus (campus cruiser service). The integration requires solving two key problems. First, the data from a new source needs to be expressed in the same data format and vocabulary as the data from the already integrated sources. Second, since we are dealing with realtime information, it is necessary to generate the corresponding RDF dynamically from the JSON data that the service returns. In this section we describe how we support the automatic modeling of sources to make it possible to rapidly integrate both static and dynamic sources into the system.
To bring in a new source or service, the rst task is to produce a model
that de nes the mapping from the source data or the data returned by the
service to a common ontology. In previous work [
Karma supports the rapid modeling of both static and dynamic sources. In
the case of a static source, a sample of the records are loaded into Karma and
then the system can proceed to generate a model based on the data. Similarly,
Karma handles dynamic sources, such as a REST-based services [
The second task is to use the model on the y to automatically produce RDF data in terms of the shared domain ontology for a given input to a service. This task is referred to as lifting since we have to take the JSON output and produce a corresponding set of RDF data that uses the vocabulary from the ontology. Karma is able to perform the lifting task automatically using the R2RML model built in the rst task. When Karma built the model in the rst task, it parsed the invocation URLs extracting the input arguments, and recorded this information in the model. In the second task, when Karma receives a service request, it automatically composes the invocation URL from the input arguments, invokes the service, receives the JSON output, uses the model to automatically map the service output into RDF in terms of the domain ontology, and nally returns the RDF.
Please note that Karma supports the modeling of tabular sources. Thus, tabular formats such as CSV and TSV can be directly loaded into Karma. Hierarchical formats such as JSON are transformed to a tabular format before modeling. 5
Having access to all sources via a Linked Data interface, we can devise a Data-Fu
program [
A Data-Fu program consists of i) facts and actions, ii) queries and iii) rules. Facts are RDF triples; actions are HTTP operations, such as HTTP GET or PUT. In our current scenario, we limit ourselves to read-only (GET) operations. Data-Fu supports conjunctive queries that either return variable bindings (select queries) or triples (construct queries). Rules consist of a conjunctive query in the antecedent and either triple patterns (for deduction rules) or actions (for interaction rules) in the consequent.
The state manipulation abstraction and the declarative speci cations describing the interplay between resources bring the following major bene ts: { Scalable execution: declarative speci cations can be automatically parallelized more easily than imperative programs. { Uniform and consistent error handling: instead of being confronted with source-speci c error messages, universal error handlers can be realized. { Substitution of resources: replacing a source only requires modeling the source in Karma in terms of the domain ontology. Such exibility is required as in networked applications the underlying base resources may become unavailable and thus may put the entire application at risk. { More exible and cleaner speci cations of interactions: the speci cations can concentrate on the business logic of the intended interaction, while the operational interaction between components can be automated due to their standardized interfaces.
The Data-Fu interpreter executes resource interactions speci ed in Data-Fu
programs. The engine holds the current state of the interactions as well as the
functionality to invoke interactions with resources as de ned in the rules. In
practice, we translate a Data-Fu program into a logical data ow network, which
is then transformed into an evaluator plan that actually implements the data ow
network. For performing the HTTP operations, we use a multi-threaded lookup
component that carries out the HTTP requests and streams the resulting triples
into the data ow network. The data ow network then processes the triples and
provides information about further HTTP requests to the lookup component.
The division into data ow network and lookup component is similar to the one
in [
The following program collects information on busses from the wrapped LA Metro API. We omit pre x declarations which can be found on pre x.cc6. First, we specify an HTTP GET request which retrieves RDF describing the bus routes. [] http:mthd httpm:GET ; http:requestURI <http://openeanwrap.appspot.com/route/> .
Then, we specify rules that trigger GET requests on :Route and geo:Point URIs. { ?route rdf:type <http://km.aifb.kit.edu/people/aha/2013/d3/cruiser#Route>. } log:implies { [] http:mthd httpm:GET ;
http:requestURI ?route . } . { ?stop rdf:type <http://www.w3.org/2003/01/geo/wgs84_pos#Point>. } log:implies { [] http:mthd httpm:GET ;
http:requestURI ?stop . } .
We also may specify deduction rules that allow for the encoding of the semantics of vocabulary terms. One can also use pre-de ned rule-sets that specify the entailment of certain OWL constructs7.
Finally, we specify a query that returns an identi er, a label, and latitude/longitude information for all of the data that ows through the network. <q1> qrl:select ( ?x ?label ?lat ?lon ); qrl:where { ?x rdfs:label ?label . ?x geo:long ?lon . ?x geo:lat ?lat . } .
The only thing left to do is to invoke the program and convert the query results to a format that our visualization toolkit understands. Thus, we request the query results in SPARQL XML Query Result format, and apply an XSLT which transforms the query results into KML for display.
The amount of data fetched during the evaluation of Data-Fu programs can range in the tens of MBs. As fetching all the data at query time is infeasible, we decouple evaluation of programs from the access to query results. The evaluation of the program can be triggered with a HTTP POST request (optionally including input data such as bounding box); the program then evaluates and the query results are updated. User agents can then GET the query results under the query's URI. One particular aspect we want to exploit in future versions are update mechanisms based on headers for cache control and expiry of documents. Thus, the interpreter can use the server-provided expiry headers to update query results automatically once the validity of one of the sources expires.
Data-Fu integration programs can be used for slowly changing sources (update intervals measured in months) and frequently changing sources (update intervals measured in seconds). Initial experiments on running Data-Fu programs in a realistic settings with real data are described in the following section. 6
We implemented a prototype data integration system to test the feasibility of our method. We rst describe the domain ontology O for the scenario, then we explain how we constructed Linked Data interfaces L to the original data sources D, and nally we show performance results for evaluating the integration programs P.
The domain ontology consists of several terms from dc, dct, foaf, geo and others. In the following we describe how we model moving objects. There is a class :Vehicle; each location reading is part of an :Observation, which connects to a :Vehicle and contains a geo:Point, and optionally additional information such as heading or a timestamp. Such a representation is su cient to query the moving objects and display the results on a map.
The types of relevant original data sources D include:
{ Static sources such as 2D maps, 3D models and point-of-interest (POI) data from les in XML, JSON or CSV, Linked Data, and web APIs. { Dynamic sources producing state updates (e.g. of moving objects or event information), continuously or periodically, possibly with additional spatial and temporal properties, through web APIs.
All sources with a Linked Data interface L support dereferencable URIs. However, we require means to query for sets of URIs with certain properties. Thus, we arrive at additional lookup or query possibilities for sources: { Keyword lookup, which returns a list of URIs matching the keyword { Point lookup, which returns a list of location URIs near the speci ed point { Bounding box lookup, which returns a list of location URIs within the bounding box
Table 1 lists properties of the sources as Linked Data L and shows the query time for retrieving the information necessary to render points on the map. The table also contains the lines of code and the number of rules per integration program for each source. One objective of our approach is that we are able to quickly add new sources: the lines of code for data integration programs P give an indication as to the e ort required to add sources.
Data Source Input Size Triples Runtime LoC Rules Marine vessels (AIS) bounding box 39 KB 224 0:01 min 33 2 Campus Cruiser - 65 KB 406 0:01 min 19 1 Crunchbase keyword 14 MB 87,477 12:08 min 31 2 Eventful keyword 55.3 MB 293,975 57:13 min 46 5 GADM bounding box 10.8 MB 68,640 7:08 min 33 2 GeoNames (Wikipedia) bounding box 2.3 MB 14,746 1:00 min 32 2 GeoNames (cities) bounding box 304 KB 2,629 0:04 min 32 2 LastFM point 10.1 MB 77,431 13:39 min 34 2 LA Metro (vehicles) - 130 KB 860 0:01 min 39 5 LA Metro (routes) - 2.5 MB 17,349 1:03 min 43 6 OpenStreetMap bounding box 8.4 MB 57,904 8:29 min 36 2 Wikimapia bounding box 8.8 MB 51,714 0:19 min 41 3
The table also shows that the Data-Fu interpreter scales from small sources (hundreds of triples) to sources of moderate size (hundreds of thousands of triples). Small sources (such as marine vessels and LA Metro vehicles) can be repeatedly polled to retrieve their current state. The number of moving objects changes depending on the time of day; on a weekday noon we have seen 56 buses and 72 vessels in the current con guration.
We have anecdotal evidence that our infrastructure is capable of much faster polling than supported by the sources. Our infrastructure can handle all LA Metro vehicles; however, we restrict the number of tracked bus lines in the prototype to avoid overloading the source. We additionally throttle lookups and restrict the number of parallel lookups to four. Future work will have to include further investigation of performance and stability of sources, and mechanisms to alleviate their load, as we had repeatedly to deal with APIs that were down.
We present initial measurements of three con gurations for transforming D to L: transformation via procedural wrappers on Google App Engine (GAE) and on a local Tomcat installation, and transformation via Karma. The Crunchbase wrapper (JSON to RDF in Java) deployed GAE has an overhead of -29 %, that is, the wrapped source is actually responding faster than the original source. Our hypothesis is that the wrapper on GAE is closer in the network (12 hops/6.1 ms measured with traceroute) than the original API (22 hops/161.5 ms), while the remaining data transfer is routed via Google's optimized internal infrastructure.The AIS wrapper (XML to RDF via XSLT) deployed on Tomcat incurs an overhead of 246 % (76 ms average for direct API access vs. 263.8 ms for wrapped access). The campus cruiser source modelled and transformed via Karma (JSON to RDF in Karma) incurs an overhead of 29 %. Further studying the overhead for transforming original sources is part of future work. 7
We survey related work in two areas: sensor systems and web systems. In these areas we di erentiate between two fundamental architectures for accessing dynamic data: pull-based (polling) and push-based (streaming) data acquisition. We distinguish between source (i.e., the site where the data originates, such as the sensor) and target (i.e., the site where data and queries are processed). We further present related work in the area of declaratively speci ed networking.
There has been a considerable amount of research in the area of streaming data, i.e. data that arrives at intervals. The goal of stream data management systems is to evaluate queries over high-velocity data, often in the context of physical sensors. Systems are able to process aggregation queries (often including query clauses with a temporal component) over time-stamped data.
With pull-based (polling) data acquisition, the target accesses the sources
in intervals and coordinates data access centrally. Pull-based access is used, for
example, in TinyDB [
We also distinguish between pull-based and push-based systems in related
work in the area of query evaluation over web sources. Polling is the
predominant mechanism used in systems based on RESTful architecture, such as the
web, and is supported in HTTP. Hartig et al. describe a system that allows for
query evaluation over Linked Data [
In contrast to the pull-based approaches, there are several proposals for
streaming query processing over RDF. Notable references in the area of Linked
Data include Linked Stream Data [
Although there are historical datasets which provide time-stamped
observations of e.g. weather phenomena, and the W3C Semantic Sensor Networks XG8
speci ed a vocabulary for describing sensors and observations, to date there is no
established protocol for streaming over the web, and the availability of streaming
sources online is scarce. Finally, there has been work on declaratively specifying
data-centric programs in a cloud setting [
We have shown the feasibility of on-the- y integration of static and dynamic sources for applications that consume Linked Data. We believe that current web architecture o ers the right abstraction and allows for the cost-e ective implementation of such systems. Linked Data already supports the integration of static data and the same mechanism can be used to achieve real-time functionality. We have shown how to provide interactive access to data, which requires to execute integration pipelines at query time within seconds for real-time sources.
We can envision to extend our research in the following directions: { Temporal aspects: in the prototype, we are concerned with showing the current state of the tracked objects. We would like to extend the temporal model to include past states of the objects, and potentially future (predicted) states. { Fault handling: although our current system is robust in terms of failing APIs (the failing source is just not displayed in the refresh cycle), we need to further investigate fault handling and error reporting. { Reducing network tra c: we plan to experiment with HTTP expires and caching mechanisms to reduce the amount of data transferred while keeping the query results up to date. 8 http://www.w3.org/2005/Incubator/ssn/XGR-ssn/