The world is ever-changing, so data portals publish live datasets derived from streams of data (events, deltas, etc.). Streams are often related to other streams by transforming their data. This leads to problems when datasets and streams are not accompanied by provenance information. Query agents often use this information to query each dataset only once, while no or inadequate provenance information forces the query agents to query the same underlying data multiple times. Once a required dataset is found the query agent can go further and discover the most fitting interface. Different interfaces determine how data is accessed.
For example in a route planning application that notifies the user about changes in routes due to construction sites, multiple interfaces can be used. A time-based interface makes it easy to track the latest changes, whereas a geospatial-based interface makes it easy to calculate whether or not a construction site will be encountered on a particular route. A SPARQL endpoint can fulfil both interfaces, but SPARQL carries high operation costs and may leave the users with
Envelope arthur.vercruysse@ugent.be (A. Vercruysse); x.sittminoo@ugent.be (S. M. Oo); pieter.colpaert@ugent.be (P. Colpaert) GLOBE https://pietercolpaert.be (P. Colpaert) Orcid 0000-0003-1586-5122 (A. Vercruysse); 0000-0001-9157-7507 (S. M. Oo); 0000-0001-6917-2167 (P. Colpaert) low availability [1]. Constantly updating a data dump is also an option, but leaves the client with high operation and bandwidth costs [2]. Linked Data Event Streams (LDES) alleviate these high costs by fragmenting the data in different web resources and linking these resources with semantic links in multiple dimensions [2], focussing on easy replication and synchronization. Each stream can be fragmented in a different way to accommodate to the needs of the users.
We set up a new interface for the Belgian street name registry and demonstrate that a query agent can find the optimal interface to execute particular queries.
DCTerms, DCAT and VoID: Exposing metadata about datasets is long established. Dublin Core Terms (DCTerms) can be used to provide basic information about resources, providing terms like title, author, description and subject [5]. Data Catalog Vocabulary (DCAT) is designed to facilitate interoperability between data catalogs published on the web [6]. DCAT also provides terms like license, which makes it possible to define a new license for an interface. The Vocabulary of Interlinked Datasets (VoID) focuses on explicitly linking datasets together on some predicate and defining subset datasets [7].
LDES: Linked Data Event Streams is a way of exposing an evergrowing set of immutable objects. These objects can be divided into fragments as HTTP resources that are linked together with the TREE specification. Fragmentations are used to provide semantic meaning to links between HTTP resources. Each HTTP resource can, for example, hold all items that start with a particular letter. A view description is used to define the meaning of the fragments and their links [2].
VoCaLS: Vocabulary for Cataloging and Linking Streams and streaming services on the web extends the ideas of DCAT with more information about streaming data [8]. The work defines a stream slightly differently than in this paper. VoCaLS focuses on streams that generate high throughput updates, this requires processors to use a windowing mechanism. In this paper, a stream is seen more broadly as a growing collection of objects, updates or otherwise.
The Ontology for Provenance and Plans (P-Plan) is an extension of the PROV-O ontology [9] created to represent the plans that guide the execution of scientific processes. P-Plan describes how plans are composed. This information can be used to keep track of provenance tree [10].
A stream in the context of the Smart Data Specification for Semantically Describing Streams (SDS) is a physical live channel that carries updates or items. A dataset can be derived from a stream as the collection of all updates or items. A physical channel can be any medium such as a Kafka stream, a WebSocket stream or even a file where updates are appended. A stream can carry any kind of data: CSV rows, mutable or immutable linked data objects, video stream chunks, etc. A stream can be derived from a transformation applied to items of a different stream. This transformation is described with p -p l a n and resides in the SDS description of the resulting stream. The stream and the transformation correspond with p : p l a n : E n t i t y and p -p l a n : A c t i v i t y respectively. This is shown as the pink part of Figure 1. The transformation links to the previous stream with the p r o v : u s e d predicate. With the power of the p -p l a n , query agents can understand how datasets are linked and what interface fits a specific query the best.
The SDS description also links to metadata about the resulting dataset with the s d s : d a t a s e t predicate. This makes modifying the datasets' metadata possible after a transformation. This is represented as the green part in Figure 1.
A s d s : S t r e a m should describe the type of objects it is carrying. First, the stream can use the s d s : c a r r i e s predicate that points to the type of the s d s : R e c o r d s it is carrying(see below). Next, the stream can describe the shacl shape with s d s : s h a p e that these members adhere to, which can only be used if the stream carries linked data objects.
Linking specific items to the correct stream is done with
Data published with Linked Data Event Streams can be partitioned or fragmented in a multitude of ways. This helps query agents resolve their queries as fast as possible whilst ingesting as little data as possible. A default fragmentation constitutes a timestamp fragmentation, this allows clients to replicate and synchronize the dataset efficiently. A substring fragmentation, on the other hand, makes autocompletion more efficient [11].
In this demo, we set up a pipeline starting from an existing LDES that exposes the registry of street names with a timestamp fragmentation. The pipeline calculates a substring fragmentation based on the name of the street and exposes a new LDES with the corresponding SDS Description and substring fragmentation. The published V i e w s can then be associated with their respective v i e w D e s c r i p t i o n as described in Listing 1.
When asking a query agent "What are the 10 latest updated street names?" starting from the newly created LDES, the query agent can derive from the SDS description that the current LDES is not suitable for this query. This query would require the query agent to request the entire LDES tree and manually find the 10 latest updates, whereas following the links from the SDS description back to the original LDES, this query would only require a few HTTP requests. One HTTP request gets the SDS description and another request gets the latest updates due to the timestamp-based fragmentation.
The SDS ontology makes it possible to add a description to a stream and the resulting dataset, this adds provenance information. The provenance links streams together and transformations applied to the streams. The SDS ontology aligns well with long-established ontologies like DCAT and P-Plan to maximize interoperability.
With the SDS description, a query agent can now automatically select the right dataset and interface based on a given query. This can make a LDES more like a querylike interface because the best fragmentation for a particular problem can be found.
SDS descriptions also make it possible to track applied transformations down to the source. This enables the user that normally only extracts data from an interface to request changes at the data source. These changes will then be propagated through the streams and result in the intended change over all interfaces.
Federated query processors, that utilize source selection based on this provenance information when selecting a dataset and interface to query the dataset, are still future work.
Funded by the Flemish government's recovery fund VSDS project: the "Vlaamse Smart Data Space". @ p r e f i x e x : < h t t p : / / e x a m p l e . o r g / n s # > . @ p r e f i x r d f s : < h t t p : / / w w w . w 3 . o r g / 2 0 0 0 / 0 1 / r d f -s c h e m a # > . @ p r e f i x l d e s : < h t t p s : / / w 3 i d . o r g / l d e s # > . @ p r e f i x p -p l a n : < h t t p : / / p u r l . o r g / n e t / p -p l a n # > . @ p r e f i x p r o v : < h t t p : / / w w w . w 3 . o r g / n s / p r o v # > . @ p r e f i x s d s :
< h t t p s : / / w 3 i d . o r g / s d s # > .