When evolving datasets are used to generate a knowledge graph, it is usually challenging to keep the graph synchronized in a timely manner when changes occur in the source data. Current approaches fully regenerate a knowledge graph in such cases, which may be time consuming depending on the data type, size, and update frequency. We propose a continuous knowledge graph generation approach that can be applied on diferent types of data sources. We describe continuously updating knowledge graph versions represented as a Linked Data Events Stream, and use an rml processor for rdf generation. In this paper, we present our approach and demonstrate it on diferent types of data such as bike-sharing, public transport timetables, and weather data. By describing entities with unique, immutable, and reproducible iris, we were able to identify changes in the original data collection, reducing the number of materialized triples and generation time. Our use-cases show the importance of mechanisms to derive unique and stable iri strategies of data source updates, to enable eficient knowledge graph generation pipelines. In the future, we will extend our approach to handle deletions in data collections, and conduct an extensive performance evaluation.
Knowledge graphs (kg) usually coexist and integrate with heterogeneous and non-rdf data
sources via mapping rules. The result of executing such mapping rules is commonly a data
dump that represents a snapshot of the kg at a certain time. However, in practice, kgs and their
sources may evolve making it dificult for data publishers to keep the kg up to date and for
consumers to stay synchronized with their latest version. Recently, Linked Data Event Stream
(ldes)1 was introduced as a mechanism for publishers to expose a stream of changes (additions,
updates and deletions) of topic-based data sources composed by data objects (a.k.a. members), as
an immutable collection [
Eficient generation of an ldes from non-rdf sources requires not only the capacity to define unique and immutable iris identifying every new version of a member, but also a mechanism that enables materializing only the members that were actually updated. Data sources may publish updates in various ways e.g. latest version of each member, only the updated members, or all members and their history. Yet, they not always provide the means to uniquely identify every update and leave it up to the consumer to handle integration.
Current kg generation approaches cannot leverage data updates. Instead, they regenerate the complete kg upon updates, even when only a subset of the data changes. This approach becomes impractical for large data sources where the cost of processing new updates grows proportionally to their update frequency and may impact services relying on the resulting kg. For example, services operating on outdated data or being unavailable during the regeneration.
We address this problem by investigating how we can uniquely and consistently (across multiple versions) identify and materialize only member updates, to avoid the costs of fully regenerating the kg. We extend an rml processor implementation to produce ldes-based kgs and demonstrate our approach through 3 use cases with varying data characteristics, e.g. size and update strategy. Preliminary tests show that our approach reduces the processing time and number of triples up to 20 times when a data source is updated compared to completely regenerating all the triples.
Through this work, we aim on reducing the generation time and size of the kg generated from dynamic and evolving data sources, while improving the overall operational behavior of dependent services and applications.
The next section discusses related work. Section 3 describes our approach. Section 4 applies our approach on 3 use cases: bike-sharing, public transport timetables, and weather data. Section 5 presents our conclusions and future work.
Mapping rules execution optimizations Optimizing mapping rules execution for rdf kg
generation is an active research domain; several approaches e.g. SDM-RDFizer [
Linked Data Event Stream (LDES) ldes2 is a collection of immutable rdf data members
such as versioned entities, observations, etc. ldes uses immutable iris [
Versioning Mechanisms to store and access versioned rdf data include the approaches
proposed by Ostrich [
In this section, we describe our approach to generate unique and reproducible iris for collection’s member updates, to enable continuous ldes-based kg generation. We discuss data collections types, how we generate iris, and expose a ldes.
Diferent data collection types exist regarding immutability and history. We identify 5 types given history availability and data immutability (Table 1).
Based on their update strategy, we consider that data collections may be: (i) immutable or (ii) mutable. A collection is immutable if all its members and their updates can be uniquely identified through data member properties e.g. timestamps or hash codes. This way, consumers can identify each update individually without comparing against previous versions. In contrast, 2https://w3id.org/ldes/specification 3https://w3id.org/tree/specification 4https://www.w3.org/TR/shacl/ 5http://mementoweb.org/guide/rfc/ 6https://git-scm.com mutable collections require consumers to compare each member against previous versions to identify if the member was updated. In which case, consumers store the new version for handling later version reconciliation.
Regarding history availability, we identify 3 diferent types of data source: (i) latest state, (ii) latest changes, and (iii) full history. Latest state collections publish the latest version of all its members on every update. Latest changes collections publish only updated members (a.k.a. delta updates). In this case, consumers must retrieve the complete collection upfront and reconcile updates over it. In both previous cases, it is up to the consumers to record history. Full history collections provide the latest state of all members, including also (some) previous versions. In such case, member versions are normally identified by means of a version ID property, making them immutable.
In this section, we introduce our approach to generate unique and reproducible iris by leveraging properties of immutable data collections or by observing properties of mutable ones.
Immutable data collections provide member properties which are renewed when a member is updated e.g. timestamps or hash codes. When generating kgs from this type of data collections, we use such properties to create unique and reproducible iris for named entities. We highlight that these properties must be unafected by external factors. For example, timestamps may be afected by timezones, in such case, the timezone ofset must also be present for the collection to remain immutable. Mutable data collections on the other hand, do not provide unique member properties across updates. Consumers must compare each data member against a previous version to identify updates. Once a member update is found, a unique property such as a timestamp or hash code needs to be externally generated to create unique and reproducible iris for named entities. Keeping track of the generated iris for each collection’s member and updates, allows to identify updates and avoid generating duplicates.
Our approach for generating unique and reproducible iris for mutable and immutable data collections is illustrated in Figure 1. We generate iris with an unique identifier for each data member updateFirst, we check if the collection is immutable or not, which is described in the rml mapping rules. If immutable, we proceed to generate an iri given that members already include at least one property to produce an iri that is unique and reproducible (Section 3.1). If the collection is mutable, we compare the data member properties with a previous version (if any) to identify if a member was updated. To achieve this, we describe in the mapping rules which properties we want to observe of each data member. If a change is detected, we generate a unique value, e.g. timestamp or hash code based on the observed properties, to uniquely identify this member update and store this new version for comparison with further updates. Leveraging these unique and reproducible iris, we can avoid generating duplicates by checking if the iri was already generated in the past or not. This way, only data members which are updated, trigger rdf materialization.
Listing 1 shows an extract of an rml mapping that produces unique iris for an mutable data collection. Subject iris are generated through a FnO described function (line 6). This function receives as input (i) the iri template (line 9); (ii) a flag indicating if at least one property of the member is unique across updates (line 12) and; (iii) a set of properties to observe to check for updates (line 15). In this example, watched properties are input in a URL parameter-like form, but this can be adjusted according to the idlab-fn:generateUniqueIRI function implementation. Resulting subject iris in this example conform to http://ex.org/prop1/prop2#timestamp. We use the generation time as timestamp in this example to make the iri unique, but hash codes can be applied here as well.
Listing 1: rml Subject Map for generating a unique and immutable iri from a mutable data collection.
We also introduce ldes Logical Target which is used to export the generated ldes-based rdf
into a file. Thanks to the modularity of rml’s Logical Target [
An ldes Logical Target (Listing 2) is a regular rml Logical Target with an additional type (ldes:EventStreamTarget). Each ldes:EventStreamTarget adds ldes specific properties to a regular Logical Target such as ldes paths (lines 6-7) or a shacl shape (line 8). An ldes Logical Target only adds metadata to the serialized output. It also inherits all the characteristics of a regular rml Logical Target7, allowing it to be written into a file, a triple store, etc.
Listing 2: ldes Logical Target definition in rml to export an ldes to a file. 1 :LDESToFile a rmlt:LogicalTarget; 2 rmlt:target [ a ldes:EventStreamTarget; 3 dcat:distribution [ a dcat:Distribution; 4 dcat:accessURL <file://data/dump.nq.zip>; ]; ]; 5 rmlt:serialization formats:N-Quads; rmlt:compression comp:zip; 6 ldes:timestampPath sosa:resultTime; 7 ldes:versionOfPath dcterms:isVersionOf; 8 tree:shape <http://example.org/shape.shacl>; .
In this section, we describe practical examples of our approach, applied on 3 use cases: (i) BlueBike8 bike-sharing availability data, (ii) NMBS9 public transport timetables, and (iii) KMI10 meteorological data with the RMLMapper as rml processor. We implemented our approach in the RMLMapper, but it can be implemented in any other rml processor since we re-use existing features such as FnO functions to generate unique and reproducible IRIs; and rml’s Logical Target for producing ldes-based kgs. Our mapping rules 11 and implementation 12 are available on GitHub.
BlueBike bike-sharing data provides a dump of all stations and currently available bikes. This data source is immutable. Its members include a data property that is unique across updates, and it follows a latest state update strategy. We mapped the data using a GBFS-based vocabulary 13 into rdf, and exported it as an ldes with our ldes Logical Target. We retrieved every minute the latest state and generated unique and reproducible IRIs with our approach (Section 3.2), by leveraging the immutable last_seen timestamp property, which is included on every data member of this collection.
NMBS is the Belgian railway agency which publishes its yearly timetable schedule as a GTFS 14 dump. This data source is mutable since there are not unique data properties across updates, and follows a latest state update strategy. Each day the schedule is republished without providing a list of updates. The GTFS dump is retrieved daily to check for updates. We define a set of observed properties that are used to asses member updates (e.g. parent_station and stop_id for GTFS stops). If an updated member is found, we generate a unique and reproducible IRI for the member by adding the current timestamp when the mapping rules were executed.
KMI is the Belgian meteorological institute, which provides measurements of their Automatic Weather Stations as a csv dump. This data source is immutable since it contains historical data about the measurements with unique identifiers. We leveraged these identifiers, namely the code property, to generate unique and reproducible IRIs.
We performed preliminary tests by retrieving dumps for each of the use cases and replayed them in the same retrieval order to verify the impact of our approach on generation time and number of materialized triples. We observed reduced generation time that is proportional to the data source size. For the smaller data sources BlueBike and KMI, the generation time is 1.1x and 1.2x faster. For the bigger data source NMBS we register a generation time 24.7x faster. In terms of number of materialized triples, we see a reduction in all cases: BlueBike 4.6x less triples, KMI 17x less triples, and NMBS 33.7x less triples; compared to fully regenerating each kg. These results are available on Github15.
11https://github.com/RMLio/RML-LDES-mapping-rules 12https://github.com/RMLio/rmlmapper-java/releases/tag/v5.0.0 13https://github.com/pietercolpaert/Blue-Bike-to-Linked-GBFS 14https://gtfs.org 15https://github.com/RMLio/RML-LDES-mapping-rules/
We see very promising results of our approach in terms of eficiency for continuous kg generation. We manage to speed up generation time up to 24x while reducing unnecessarily materialized triples up to 33.7x. Our approach relies on existing rml features such as FnO functions, which facilitates its implementation on existing engines. Furthermore, our implementation provides out-of-the-box publishing of kg updates in the form of an ldes, which facilitates data synchronization tasks for consumers. In the future, we will extend our approach to handle deletions and perform a more exhaustive performance evaluation.