This is the report of our participation in the KGCW2023 Challenge @ ESWC 2023 with our RDFProcessingToolkit/Sansa system which won the “fastest” tool award. The challenge was about the construction of RDF knowledge graphs from RML specifications with varying complexity in regard to the mix of input formats, characteristics of the data and the needed join operations. We detail how we integrated our tool into the provided benchmark framework. Thereby we also report on the issues and shortcomings we encountered as a base for future improvements. Furthermore, we provide an analysis of the data measured with the benchmark framework.
Accompanying the challenge, a pipeline tool3 was provided. The benchmark tool is built on Python, JSON, and Docker. It was strongly encouraged to use the tool and of course, the promise of its provisioning is to ease the evaluation of the system under test and have a common ground for comparing results.
After installing the benchmark tool, first we had some issues getting it to work. One major
point of critique, which was also shared by other challenge participants, is that the provided
version of the benchmark tool proceeded to delete all Docker containers on the system, even
those that were not related to the challenge. For this reason, we had to patch our own fork
of the benchmark tool such that only containers that were also created by it were removed.
Additionally, the included MySQL component caused the following three noteworthy troubles:
(
Then, it was needed to integrate RPT/Sansa. The steps involved were:
1. Creation of a Docker image for RPT. As RPT already ships with several packaging options,
including Docker,4 there was no additional work needed.
2. Writing of a Python file that connects RPT to the benchmark tool. The file is
bench_executor/rpt.py and was suficiently easy to create by copying and adapting
the rmlmapper.py file. However, there were some dificulties: (
The challenge dataset comes with 63 metadata.json files which use RMLMapper as the reference to implement the necessary steps. At first, we pondered if a search and replace operation on these files would be suficient to run RPT, however it is easy to make mistakes in the process, which would have been detrimental to our challenge run. So instead we examined all files and identified the common patterns. On this basis, we reverse-extracted two YAML templates from those 63 files, one for each major part of the challenge. Then we continued to implement a template-to-metadata.json program which generates all 63 files from those two templates as required.5 3https://github.com/kg-construct/challenge-tool 4Maven artefact rdf-processing-toolkit-pkg-docker-cli 5https://github.com/SimonBin/kgc-challenge-tool-template
Because we also wanted to be able to run and compare multiple tools using the benchmark tool, we furthermore changed the pipeline filename from metadata.json to my-tool.json, and added a command line switch to the benchmark tool to change the filename that it will read.
The RML mappings provided in the challenge sufered from the following issues, which we
resolved using SPARQL Update queries: (
Next, we also replaced R2RML in the mappings with RML and changed the table names to the CSV file names, as RPT / Sansa did not support relational databases (RDB) at the time of the challenge. It is noteworthy that this adaption was also made by at least one more participant. In this case, separate tasks for RDB and CSV would have been desirable.
We verified the RDF output of our tool by comparing it to the ground truth provided by [ 3]. For this purpose, we harmonised our output and the reference data as sorted n-quads and performed a simple line-based white-space ignoring comparison using the diff command. Although the number of triples matched up, there were lines that difered. Manual inspection revealed that there were issues with the ground truth: • Many numbers in the ground truth are replaced with 999.999, one example from GTFSMadrid-Bench, scale_100: <http://transport.linkeddata.es/madrid/metro/shape_point/00000000000000000001-1000803> <http://www.w3.org/2003/01/geo/wgs84_pos#long> "999.999999999999999"ˆˆ<http://www.w3.org/2001/XMLSchema#double> .
Correct answer: "3971182"ˆˆ<http://www.w3.org/2001/XMLSchema#double> • Data types were present in the ground truth which were not specified in the mapping ifles. One example from GTFS-Madrid-Bench, heterogeneity_files: <http://transport.linkeddata.es/madrid/metro/trips/000000000000000009kc> <http://vocab.gtfs.org/terms#direction> "1"ˆˆ<http://www.w3.org/2001/XMLSchema#integer> .
Correct answer: "1" (as a string)
There is one more caveat with the GTFS benchmark: wrong use of data types. The arrival_time and departure_time are mapped to xsd:duration, however their values are not a valid duration. (Example: 0000000000000000006l; valid duration example: PT2M10S). This causes systems that validate RDF terms, such as ours which is based on Apache Jena, to spend a significant amount of time emitting warning messages. Thus, a recommendation for the future would be to improve 6https://github.com/oeg-upm/gtfs-bench/issues/142#issuecomment-1453784200 7https://rml.io/specs/rml/#examples the benchmark with only well-formed RDF terms and/or have dedicated tasks for mappings that produce malformed ones.
In this section we present the results which we obtained for RPT/Sansa using the benchmark
tool. Unfortunately, for this challenge it was not possible to evaluate all participating tools on
the same hardware. Each participant executed their own tool on their own system. Our system
was an AMD Ryzen 9 5950X with 128 GB RAM, of which Java was allowed to use 50% of the
RAM. The benchmark tool generated 126 CSV files when giving the command ./exectool
stats. To that end, we wrote a program to first extract these files in an automated way and then
to draw plots of the measured values of RPT. Otherwise it would be laborious and error-prone
to create the charts. We also calculated the number of triples resulting from the mapping as well
as the input sizes of the source files processed by the mapping (these values are not present in
the CSV files created by the benchmark tool). For a comparison of RPT to other systems please
refer to our system paper [1]. In the following we only show selected plots. All plots, as well as
the raw data, can be found in the RPT supplements repository.8
How to read the plots. Each of the following plots captures all of the following information.
In order to assist the reader, we added circled numbers to Figure 1 which correspond to the
numbers of this list:
1. The challenge was organised into multiple directories with diferent tasks. The parent
directory/task is displayed on the left outer legend, the left y-axis shows the
directory/different parameter configurations for this task.
2. The thick bar next to the parameter configuration is the main bar. It records the run-time
of RPT. The total run-time in seconds is also printed at the end of the bar, and the run-time
is also displayed on the bottom x-axis.
3. Each task “pipeline” can be configured to consist of multiple steps. We configured RPT to
use six steps, the duration of each step corresponding to the diferent coloured segments on
the main bar. (
The memory consumption is also displayed on the top x-axis. Note, that the measurement has very limited informative value because the benchmark tool captures the initial virtual memory claimed by the Java Virtual Machine (JVM, 28 GiB) and not the actually used memory. We have no reason to suspect that less memory would be a problem for RPT. 8https://github.com/AKSW/RdfProcessingToolkit-Resources/tree/main/2023-05-28-KGCW-at-ESWC 6. and 7. The small bar above the main bar and the small bar below the main bar are to be read from top to bottom. They display the relative size of the input data on top and the relative size of the output triple count on the bottom of the main bar (blue). In the case of heterogeneous input formats, the top bar is colour coded for the size of the XML, JSON and CSV input. Note that while the raw numbers are absolute, the plot only gives a relative perception of the input and output sizes, with a size of 0% on the far left.
The tasks join 1-1, join 1-N, join 1-10, join N-1, and join 10-1 (which test scaling the number of joins) as well as empty-values (which test the handling of empty/unbound values in the dataset) do not exhibit much variation. RPT requires roughly 20 seconds to complete them, with a CPU time scale of × 4.4. (Of these, approx. 12 seconds are start-up overhead of Docker/the system.) None of the parameter variations seem to have much influence here, which might suggest the data/size/test case is too small for RPT. For the tasks join N-M, with N, M ̸= 1, we can detect a slight increase in run-time depending on the number of joining triples and thus also the resulting triple count (see Figure 1). For the duplicated-values, RPT’s performance slightly decreases the fewer duplicates there are because more time needs to be spent on writing. This is not observed for empty-values, because there are always 100 000 unique id values whereas in the duplicated-values task % of the IDs are duplicates. (See supplements for figures.) 5 memory consumption in GiB 28 32 36 40 44 48 52 56 60 64 1 5 5 n i o j , M N n i o j 100% 75% 50% csv input size 6 fix sparql opt.
map step 3 3 10
15 20 run-time in sec. 50% —— i/o size —— 100% 25
The mappings task sought to evaluate the balance between multiple triple maps (TM) and property-object maps (POM). Before running the task, we assumed that the resulting graphs should be equivalent and thus the run-time of RPT identical, since the source CSV and the triple counts are identical. However, the mappings provided are quite diferent. In the case of 15TM 1POM there are 1.5 mio. entities with exactly the same property :p1 generated whereas in the 1TM 15POM case, we have only 100 000 entities with 15 properties each. Hence for 15TM 1POM the distinct operation is more expensive and cannot be parallelised well, as is evident in Figure 2. 28.52 21.07
files mixed tabular mem 160 mem xml json
csv input size fix xml sparql opt.
The second part of the challenge was the GTFS-Madrid-Bench. We have also used this benchmark in [1], but we did not test RPT with heterogeneous GTFS Bench (n.b. there seems to be a mismatch between the description of the challenge and the actual content, we found the composition to be: csv+sql (tabular), xml,json,csv+sql (mixed), xml,json,csv (files), and xml,json (nested)). From Figure 3 we can see that CSV (tabular) has a much smaller input size. This benefits RPT two-fold, making it the fastest configuration of this task group. JSON files take the
34.73 91.74
2,500 3,000
3,500 Sansa ARQ mem × 5.8 mem triples(log) y t i e n e g o r e t e h , h c n e B S F T G 1000 100 10 1 100 csv input size(log) fix
files mixed tabular 28
32 129.67 majority of the input data for all the other cases. In the nested configuration there is three times more XML than in the other two cases. Here, we can also observe that our XML processing time (second step) takes 6.5 seconds as compared to 4.2 seconds or 2.9 seconds (when no XML is present; start-up overhead). Additionally, with this data size the parallelism can be utilised, leading to a CPU time scale of × 16.
When scaling the GTFS Bench, we can see RPT shine. The GTFS Bench can generate input data for multiples of 395 953 output triples. For scale 1000 that amounts to almost 400 mio. triples. The performance of RPT can be seen in Figure 4. We changed to a logarithmic scale (x-axis) to make it easier to compare diferences in the order of magnitude. Similar to the scaling of records, the GTFS Bench at scale 100 takes 91.7 seconds, and at ten times it takes 727 seconds (an increase of only × 8). The parallelism is also increased once more from a CPU time scale of × 18 for GTFS scale 100 to × 25 for GTFS scale 1000. That means in number of CPU seconds, for a ten-fold increase in output triples we have a still satisfactory eleven-fold increase.
Figure 5 shows a bonus plot comparing the execution time of RPT/Sansa to that of RPT/ARQ, another execution engine which is running on the same infrastructure as RPT/Sansa but with Jena ARQ instead of Apache Spark as the execution engine. This is a single-process execution engine without distributed/parallel computation.
We initially expected that all contenders in the challenge would create configurations for the benchmark tool such that Docker containers could be easily run with diferent Knowledge Graph Construction tools on the same hardware. However, this efort has yet to happen. It would have been great if the benchmark tool were capable of generating a statistics report similar to the one we described automatically (ideally even with support for comparing multiple systems). So far the benchmark tool only collected data in a set of CSV files but all further interpretation had to be done individually by each participant. Finally, as apparent in the results, RPT/Sansa executes most tasks in around 20 seconds. As such, by comparing RPT only to itself there are not many interesting diferences to be seen from which insights could be derived besides that RPT executes those tasks in a stable and reliable way. Still, for future challenge runs, it may be worthwhile to refine the tasks further such as by scaling them up to larger sizes in order to see whether this leads to significant diferences.
The authors acknowledge the financial support by the German Federal Ministry for Economic Affairs and Energy in the project CoyPu (project number 01MK21007A) and by the German Federal Ministry of Education and Research in the project StahlDigital (project number 13XP5116B).