This paper presents the results of the performance track of the Knowledge Graph Construction Workshop 2024 Challenge with RMLStreamer, an RML mapping engine that processes all data in a streaming fashion. On mappings without joins, RMLStreamer scales well regarding execution time and CPU usage, while maintaining a constant memory usage. To optimize the processing of the joins, we added RML-view-to-CSV as a first step to our knowledge graph construction pipeline. RML-view-to-CSV is a proof-of-concept implementation for RML Logical Views, i.e. flattened, source format-agnostic views over one or more existing data sources. RML-view-to-CSV can additionally rewrite referencing object maps as logical views, before it materializes the logical views as CSV files. The combination of RML-view-to-CSV and RMLStreamer emerges as an eficient approach, showcasing the potential of modular mapping engines that delegate each task to the most suitable framework.
CEUR ceur-ws.org
©
2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings htp:/ceur-ws.org ISN1613-073
CHALLENGE INPUT:
RML mapping CHALLENGE INPUT: source data
RML mapping without joins materialized joins
(CSV files) RML-view-to-CSV
RMLStreamer
OUTPUT KGC pipeline: RDF knowledge graph
In this paper, we present the results of the performance track for RMLStreamer [
Section 2 describes the components of our knowledge graph construction pipeline. Section 3 discusses the setup used to execute the challenge’s experiments. We present our results in Section 4 and our conclusion in Section 5.
Our knowledge graph construction pipeline (Figure 1) consists of two components: (i)
RMLStreamer executes the RML mapping rules in a streaming fashion [
RMLStreamer executes RML mapping rules to generate high quality Linked Data from multiple originally (semi-)structured data sources in a streaming way. It handles big input files and continuous data streams like sensor data without consuming more memory when the input data size increases. It leverages Apache Flink to scale vertically across multiple CPU cores and horizontally across multiple machines. In the challenge, we use RMLStreamer version v2.5.0 with an embedded Flink version in a Docker container5.
The challenge results of 2023 show that, on mapping tasks without joins, RMLStreamer
scales well regarding execution time and CPU usage, while maintaining a constant memory
usage [
However, joins can significantly prolongue the execution time and increase the size of the output of RMLStreamer, as RMLStreamer does not eliminate self-joins or duplicates. As a result, RMLStreamer needs more than three hours to execute the first scale of the GTFS-Madrid-Bench, generating an output of 105 GB. Therefore, we delegate the execution of joins to RML-view-to-CSV. 4https://github.com/RMLio/rml-view-to-csv/ 5https://zenodo.org/records/7998156
RML-view-to-CSV is a proof-of-concept implementation for RML Logical Views, a new
RML module that is still under development. In our previous paper [
RML Logical Views6 allow specifying a logical view: a flattened, source format-agnostic view over one or more existing data sources. A view over multiple data sources can be created by joining a logical view with other logical views.
RML-view-to-CSV resolves logical views as a first step in the KGC pipeline. It includes an option to delegate the execution of joins expressed in triples maps to logical views. First, RML-view-to-CSV identifies and optimizes redundant self-joins, and eliminates the remaining referencing object maps from the mapping, replacing them by equivalent logical views. Afterwards, it materializes the logical views as CSV files. During this process, RMLview-to-CSV also takes the related triples maps into account, eliminating redundant fields and duplicate logical iterations. Finally, it rewrites the mapping accordingly, replacing the logical views as logical sources over the materialized logical views.
At this moment, RML-view-to-CSV supports one nested source format (JSON) and one tabular source format (CSV). Slight adaptations to the experiment setup were needed to overcome this limitation.
The KGCW 2024 Challenge provides CSV files as source data, mapping files, queries, baseline results (i.e. the expected set of triples and query results), an example pipeline based on the MySQL, RMLMapper, and Virtuoso for reaching those results, and a tool for executing the example pipeline.
The following adaptations were made to the provided experiments to enable execution with our KGC pipeline. (i) In the provided end-to-end pipelines, the CSV files are loaded into a relational database. As RML-view-to-CSV does not support SQL (yet), we used the CSV files directly to construct the knowledge graphs and adapted the mapping ifles accordingly. (ii) As RML-view-to-CSV does not support XML (yet), we replaced the XML files in the GTFS-Madrid-Bench heterogeneity experiments by JSON files. We conducted two heterogeneity experiments: one experiment with only JSON source data and one experiment with 50 % CSV and 50 % JSON source data, both on scale 100. (iii) We added a condition to the mapping files to recognize the string NULL in the provided CSV files as an empty value. The GTFS experiments were executed with our KGC pipeline as shown in Figure 1. For the KGC parameters experiments without joins, we skipped the preprocessing step with RML-view-to-CSV, as it is a redundant step for these experiments. For the KGC parameter experiments with joins, we tested both the pipeline with and without RML-view-to-CSV as preprocessor, to measure also the impact of RML-view-to-CSV for those experiments. 6https://github.com/kg-construct/rml-lv
We compared our experiments’ results to ensure that our output is correct with respect to the baseline results of the challenge. For the first part of the challenge (KGC parameters), where the output of RMLStreamer is not loaded into a triples store, we deduplicated the output results as RMLStreamer cannot eliminate duplicates by itself. After deduplication, we compared the number of triples to the baseline results of the challenge. For the second part of the challenge (GTFS-Madrid-Bench) we compared the number of query results to the baseline.
All experiments were executed on the virtual machine provided by Orange, with following specifications: 4 vCPUs, 16 GB RAM and 140GiB SSD storage running on Ubuntu 22.04.3 LTS. The challenge execution tool configures the Java heap space to 50 % of the available memory. All experiments were performed 5 times, and the experiment with the median of the measurements is reported. All files needed to reproduce the conducted experiments are available on Zenodo7.
In Figure 2 and Table 1 we included the measured execution time, CPU time, and maximal memory usage of the knowledge graph construction pipeline for selected experiments. The complete overview of the results, as it was submitted to the KGCW 2024 Challenge, is available on Zenodo8.
First, we verify how RMLStreamer behaves when the size of the expected output increases. This is best illustrated by the GTFS-Madrid-Bench scale experiments. Figure 2a and Figure 2b show that both RML-view-to-CSV and RMLStreamer scale towards a linear trend. Note that the reported metrics include the startup time of RML-view-to-CSV and RMLStreamer (no separate measurements in the challenge execution tool), that is independent of data size and has a higher impact on the lower scales. The execution time and CPU time increases with a factor of ten, the same factor as the data size for higher scales. The peak RAM memory (fig. 2c) measured is similar for all scales when using RMLStreamer. RMLStreamer has a constant memory usage independent of the data size, because it processes everything in a streaming way. This ensures a stable performance independent of the data size. As long as there is space to store the output, RMLStreamer can continue its knowledge graph construction process. For RML-view-to-CSV we note an increasing use of memory for the higher GTFS scales. Nevertheless, it was still able to handle scale 1000 without reaching the memory limitations of the provided hardware.
The measurements for the KGC parameter experiments confirm these observations (Table 1 Section 1). RMLStreamer shows linear scaling of execution time and CPU usage, proportional to the size of the input data, in combination with a constant memory usage.
Second, we evaluate the impact of the format of the data input. Replacing the CSV source data by JSON data increases the execution time of RMLStreamer with a factor of two. The diference in execution time for RMLStreamer is the consequence of RMLStreamer chunking CSV files and processing the chunks in parallel. This is not the 5.766 5.956 10.000 1 10
100
Execution time (s) (a) Linear trend: the execution time of both
RML-view-to-CSV and RMLStreamer increases with the same factor as the data size for higher scales. Execution time (s)
CPU time (s) Peak RAM memory (GB) 7 18 (b) Linear trend: the CPU usage of both
RML-view-to-CSV and RMLStreamer increases with the same factor as the data size for higher scales. (c) The memory consumption of RML-viewto-CSV increases, while RMLStreamer has a constant memory usage independent of data size. (d) When using JSON data as source instead of CSV data, the performance impact is higher on RML-view-to-CSV than on RMLStreamer. case for the JSON formats yet. The performance impact of nested data is higher for RML-view-to-CSV. The execution time and CPU usage of RML-view-to-CSV increases with a factor of ten, and its memory consumption with a factor of three (fig. 2d).
Third, we investigate the impact of duplicates and empty values in the input data. As RMLStreamer does not eliminate duplicates and the duplicate tests all start with the same amount of input data, there is no noticeable diference in performance between the experiments with and without duplicates, whilst a duplication elimination could result in a much better performance for tests with duplicates (Table 1 Section 2). In the source data of the experiments with empty values (Table 1 Section 3), this string NULL is representing an empty value in a CSV file. We had to add a condition to the mappings of those experiments to recognise this string as an empty values. The execution of this condition increased the execution time and CPU usage with 50% when all rows contain empty values.
Last, we comment on the KGC parameter experiments that include joins (Table 1 Section 4). We executed these experiments without and with RML-view-to-CSV as preprocessor and noticed that RML-view-to-CSV reduced the total execution time and CPU usage with respectively 20% and 39% for the experiments containing most joins.
The KGCW 2024 Challenge results confirms the observations of the KGCW 2023 Challenge: RMLStreamer has a linear scaling of execution time and CPU usage, proportional to the size of the input data, while maintaining a constant memory usage. Scalability is the main strength of RMLStreamer.
The main weakness of RMLStreamer is its ineficient implementation of join operations (e.g. GTFS-Madrid-Bench experiments with joins cannot be handled properly by RMLStreamer). When delegating this task to RML-view-to-CSV as a preprocessor, we resolve this weakness and build a reliable and performing knowledge graph construction pipeline.
The challenge results reveal of slower performance of RML-view-to-CSV when handling nested data. As RML Logical Views enable the flattening of nested data, the implementation of RML-view-to-CSV is processing all JSON fields separately. A more eficient implementation for nested data sources is a challenge for future implementations, after the RML Logical View specification is finalized.
Comparing to the results of KGCW Challenge 2023 [
At the moment of writing, we have no insight in the results of the other engines participating in the performance track of the KGCW Challenge 2024. We are looking forward to the comparison of the challenge results.
The described research activities were supported by SolidLab Vlaanderen (Flemish Government, EWI and RRF project VV023/10), and the European Unions Horizon Europe research and innovation program under grant agreement no. 101058682 (OntoDESIDE).