Data publishers can provide multiple interfaces per dataset. Each interface has its own merits and drawbacks, SPARQL endpoints are expensive to host and clients find it dificult to work with static data dumps. Furthermore, query agents can only select the most fitting interface and dataset if provenance information is provided. In this paper, we introduce the Smart Data Specification for Semantically Describing Streams (SDS) to annotate dataset interfaces with provenance information, describing the consumed stream and the applied transformations on that stream. We focus on Linked Data Event Streams that can publish the same dataset with diferent fragmentations and demonstrate a pipeline that transforms a LDES and publishes the data with a diferent fragmentation as described in the accompanying provenance information. The SDS vocabulary is built upon the DCAT-AP, LDES and P-Plan vocabularies. In future work, we will create a source selection strategy for federated query processors that take into account this provenance information when selecting a dataset and interface to query the dataset.
The world is ever-changing, so data portals publish live datasets derived from streams of data (events, deltas, etc.). Streams are often related to other streams by transforming their data. This leads to problems when datasets and streams are not accompanied by provenance information. Query agents often use this information to query each dataset only once, while no or inadequate provenance information forces the query agents to query the same underlying data multiple times. Once a required dataset is found the query agent can go further and discover the most iftting interface. Diferent interfaces determine how data is accessed.
For example in a route planning application that notifies the user about changes in routes due
to construction sites, multiple interfaces can be used. A time-based interface makes it easy to
track the latest changes, whereas a geospatial-based interface makes it easy to calculate whether
or not a construction site will be encountered on a particular route. A SPARQL endpoint can
fulfil both interfaces, but SPARQL carries high operation costs and may leave the users with
LGOBE
(P. Colpaert)
low availability [
We set up a new interface for the Belgian street name registry and demonstrate that a query agent can find the optimal interface to execute particular queries.
DCTerms, DCAT and VoID: Exposing metadata about datasets is long established. Dublin
Core Terms (DCTerms) can be used to provide basic information about resources, providing
terms like title, author, description and subject[
LDES: Linked Data Event Streams is a way of exposing an evergrowing set of immutable
objects. These objects can be divided into fragments as HTTP resources that are linked together
with the TREE specification. Fragmentations are used to provide semantic meaning to links
between HTTP resources. Each HTTP resource can, for example, hold all items that start with
a particular letter. A view description is used to define the meaning of the fragments and their
links[
VoCaLS: Vocabulary for Cataloging and Linking Streams and streaming services on the web
extends the ideas of DCAT with more information about streaming data[
P-Plan and PROV-O: The Ontology for Provenance and Plans (P-Plan) is an extension of
the PROV-O ontology [
A stream in the context of the Smart Data Specification for Semantically Describing Streams (SDS) is a physical live channel that carries updates or items. A dataset can be derived from a stream as the collection of all updates or items. A physical channel can be any medium such as a Kafka stream, a WebSocket stream or even a file where updates are appended. A stream can carry any kind of data: CSV rows, mutable or immutable linked data objects, video stream chunks, etc.
A stream can be derived from a transformation applied to items of a diferent stream. This transformation is described with p - p l a n and resides in the SDS description of the resulting stream. The stream and the transformation correspond with p : p l a n : E n t i t y and p - p l a n : A c t i v i t y respectively. This is shown as the pink part of Figure 1. The transformation links to the previous stream with the p r o v : u s e d predicate. With the power of the p - p l a n , query agents can understand how datasets are linked and what interface fits a specific query the best.
The SDS description also links to metadata about the resulting dataset with the s d s : d a t a s e t predicate. This makes modifying the datasets’ metadata possible after a transformation. This is represented as the green part in Figure 1.
A s d s : S t r e a m should describe the type of objects it is carrying. First, the stream can use the s d s : c a r r i e s predicate that points to the type of the s d s : R e c o r d s it is carrying(see below). Next, the stream can describe the shacl shape with s d s : s h a p e that these members adhere to, which can only be used if the stream carries linked data objects.
Linking specific items to the correct stream is done with s d s : R e c o r d objects. An s d s : R e c o r d points to the data (s d s : p a y l o a d ) and the corresponding stream (s d s : s t r e a m ). These small objects make it possible for multiple streams to use the same channel. Each transformation can thus push s d s : R e c o r d objects and leave the original stream intact. A stream of immutable objects can still be transformed, for example, to calculate a hash or add a fragment id to the s d s : R e c o r d object. The yellow part of Figure 1 gives a visual overview of s d s : R e c o r d .
Data published with Linked Data Event Streams can be partitioned or fragmented in a multitude
of ways. This helps query agents resolve their queries as fast as possible whilst ingesting as
little data as possible. A default fragmentation constitutes a timestamp fragmentation, this
allows clients to replicate and synchronize the dataset eficiently. A substring fragmentation,
on the other hand, makes autocompletion more eficient[
In this demo, we set up a pipeline starting from an existing LDES that exposes the registry of street names with a timestamp fragmentation. The pipeline calculates a substring fragmentation based on the name of the street and exposes a new LDES with the corresponding SDS Description and substring fragmentation. The published V i e w s can then be associated with their respective v i e w D e s c r i p t i o n as described in Listing 1.
When asking a query agent “What are the 10 latest updated street names?” starting from the newly created LDES, the query agent can derive from the SDS description that the current LDES is not suitable for this query. This query would require the query agent to request the entire LDES tree and manually find the 10 latest updates, whereas following the links from the SDS description back to the original LDES, this query would only require a few HTTP requests. One HTTP request gets the SDS description and another request gets the latest updates due to the timestamp-based fragmentation.
The SDS ontology makes it possible to add a description to a stream and the resulting dataset, this adds provenance information. The provenance links streams together and transformations applied to the streams. The SDS ontology aligns well with long-established ontologies like DCAT and P-Plan to maximize interoperability.
With the SDS description, a query agent can now automatically select the right dataset and interface based on a given query. This can make a LDES more like a querylike interface because the best fragmentation for a particular problem can be found.
SDS descriptions also make it possible to track applied transformations down to the source. This enables the user that normally only extracts data from an interface to request changes at the data source. These changes will then be propagated through the streams and result in the intended change over all interfaces.
Federated query processors, that utilize source selection based on this provenance information when selecting a dataset and interface to query the dataset, are still future work.
Funded by the Flemish government’s recovery fund VSDS project: the “Vlaamse Smart Data Space”. @ p r e f i x e x : < h t t p : / / e x a m p l e . o r g / n s # > . @ p r e f i x r d f s : < h t t p : / / w w w . w 3 . o r g / 2 0 0 0 / 0 1 / r d f - s c h e m a # > . @ p r e f i x l d e s : < h t t p s : / / w 3 i d . o r g / l d e s # > . @ p r e f i x p - p l a n : < h t t p : / / p u r l . o r g / n e t / p - p l a n # > . @ p r e f i x p r o v : < h t t p : / / w w w . w 3 . o r g / n s / p r o v # > . @ p r e f i x s d s : < h t t p s : / / w 3 i d . o r g / s d s # > . e x : M y L D E S a l d e s : E v e n t S t r e a m ;
d c a t : t i t l e ” A n e x a m p l e L D E S ” . # O n e f r a g m e n t a t i o n i s b a s e d o n a n e x i s t i n g L D E S e x : B a s i c F r a g m e n t a t i o n a t r e e : V i e w D e s c r i p t i o n ; d c a t : e n d p o i n t U R L < / b a s i c > ; # A r o o t n o t e f r o m w h i c h y o u c a n a c c e s s a l l m e m b e r s d c a t : s e r v e s D a t a s e t e x : M y L D E S ; # t h e L D E S # T h i s v i e w D e s c r i p t i o n w a s c r e a t e d b y i m p o r t i n g t h i s s t r e a m l d e s : m a n a g e d B y s d s : L D E S S t r e a m . # T h e o t h e r f r a g m e n t a t i o n b u c k e t i z e s t h e m e m b e r s b e f o r e p u b l i s h i n g e x : S u b s t r i n g F r a g m e n t a t i o n a t r e e : V i e w D e s c r i p t i o n ; d c a t : e n d p o i n t U R L < / s u b s t r i n g > ; # A r o o t n o t e f r o m w h i c h y o u c a n a c c e s s a l l m e m b e r s d c a t : s e r v e s D a t a s e t e x : M y L D E S ; # t h e L D E S # T h i s v i e w D e s c r i p t i o n w a s c r e a t e d b y i m p o r t i n g t h i s s t r e a m l d e s : m a n a g e d B y s d s : B u c k e t i z e d S t r e a m . e x : I m p o r t L D E S a p - p l a n : A c t i v i t y ; r d f s : c o m m e n t ” R e a d s c s v f i l e a n d c o n v e r t s t o r d f m e m b e r s ” ; p r o v : u s e d < h t t p s : / / s m a r t d a t a . d e v - v l a a n d e r e n . b e / b a s e / g e m e e n t e > . e x : L D E S S t r e a m a s d s : S t r e a m ; p - p l a n : w a s G e n e r a t e d B y e x : I m p o r t L D E S ; s d s : c a r r i e s s d s : M e m b e r . e x : B u c k e t i z e S t r a t e g y a l d e s : B u c k e t i z e S t r a t e g y ; l d e s : b u c k e t T y p e l d e s : S u b s t r i n g F r a g m e n t a t i o n ; # z e g t a a n d e c l i e n t d a t i k e e n t i m e s t a m p f r a g m e n t a t i o n t r e e : p a t h r d f s : l a b e l ; l d e s : p a g e S i z e 5 0 . e x : B u c k e t i z e S t r e a m a p - p l a n : A c t i v i y ; r d f s : c o m m e n t ” E x e c u t e a s u b s t r i n g b u c k e t i z a t i o n o n t h e i n c o m i n g s t r e a m ” ; p r o v : u s e d e x : L D E S S t r e a m , e x : B u c k e t i z e S t r a t e g y . e x : B u c k e t i z e d S t r e a m a s d s : S t r e a m ; p - p l a n : w a s G e n e r a t e d B y e x : B u c k e t i z e S t r e a m ; s d s : c a r r i e s s d s : M e m b e r .
Listing 1: RDF sample creating two LDES Views from diferent Streams