The amount of data being generated in recent years has increased drastically. Thus, a unified schema must be defined to bring multiple data sources into a single format. For that reason, the use of knowledge graphs has become much more commonplace. When creating a knowledge graph, diferent parameters afect the creation process, like the size and heterogeneity of the input data and the complexity of the input mapping. Multiple knowledge graph creation engines have been developed that handle these parameters diferently. Therefore, a benchmark is needed to be defined to evaluate the performance of these engines. KGCW 2023 Challenge dataset presents a wide array of test cases to discover each engine's strengths and weaknesses and determine which engine is best suited for each case. This work reports the results of evaluating the performance of SDM-RDFizer while using this dataset.
CEUR Workshop Proceedings RML mappings with diferent level of complexity and data sources of diferent sizes, but they do not cover all possible parameter configurations that an RML engine should cover. Therefore, KGCW 2023 Challenge dataset was developed to cover as many test cases as possible. These test cases contain joins of multiple levels of complexity, data sources of multiple sizes in terms of the number of records and properties, the number of triples maps (TMs), diferent data duplicate rates, and cases with empty values. This challenge dataset aims to discover the strengths and weaknesses of existing state-of-the-art engines and determine which engines are suitable for which test cases. This work will present the results of evaluating the performance of SDMRDFizer with the KGCW 2023 Challenge dataset.
This paper is organized into three additional sections. Section 2 defines everything regarding SDM-RDFizer, like what techniques, data structures, and physical operators it has for optimizing the KG graph creation process. Section 3 reports on the results of the empirical evaluation, including the definition of the dataset and the corresponding analysis. Finally, Section 4 illustrates the conclusions and future steps for SDM-RDFizer.
SDM-RDFizer [
The Triples Map Planning (TMP) module reorders TMs so that the most selective mappings are evaluated first; meanwhile, non-selective mappings are executed last. In other words, the TMs that have the most overlap with other TMs are executed first. TMP organizes the TMs and data sources so that the number of triples stored in memory is kept at a minimum. Therefore, when a triple is generated, the amount of comparison needed to determine if it is a duplicate is reduced and the KG creation process is quicker. TMP defines two data structures: Organized Triples Maps List and Predicate List.
Organized Triples Maps List (OTML) groups the TMs by their data source. OTML is only used when files are used as data sources (e.g., CSV, JSON and XML). During the TMP phase, TMs are classified based on the logical data source format (i.e., CSV, JSON, and XML). Afterward, they are grouped by their data source; thus, a data source is opened once to execute all the TMs. Predicate List (PL) groups TMs by their predicates by creating a list of TMs associated with a particular predicate. PL has two purposes; the first is to determine when the Predicate Tuple Table (PTT) associated to a certain predicate can be flushed and the second is to organize the OTML. Each time a TM is evaluated, it is removed from the list of the associated predicate; when the list is empty, the corresponding PTT can be flushed.
The Triples Map Execution (TME) module generates the KG; it follows the order established by the TMP module when executing the TMs. This module presents multiple data structures that optimize diferent aspects of the KG creation process, like duplicate removal, join execution, and data compression. Multiple novel operators are defined to transform diferent types of TMs. 2.2.1. Data Structures Dictionary Table (DT) encodes each RDF resource generated during the KG creation process with an identification number and each identification number is encoded in base 36. Therefore, triples are stored not as a series of resources but as numbers.
Predicate Tuple Table (PTT) stores all the triples generated so far for a predicate . PTTs correspond to hash tables where the hash key of an entry corresponds to the encoding of the subject and object of a generated RDF triple, and the value of the entry is the encoding of the RDF triple. The subject and object are stored with their corresponding identification number. The purpose of a PTT is duplicate removal; each triple generated is compared to its corresponding PTT. If the triple exists in the PTT, then it is discarded. If the triple does not exist in the PTT, it is added to the PTT and the KG.
Predicate Join Tuple Table (PJTT) stores the resulting subjects that come from executing a join condition. It is implemented as an index hash table to the data source of the parent triples map in a join condition. A PJTT key corresponds to encoding each value(s) of the attributes in the join condition. The value of a key in a PJTT corresponds to encoding the subject values in the data source of the parent TMs associated with the values of the attributes in the hash key. 2.2.2. Physical Operators Simple Object Map (SOM) generates an RDF triple by performing a simple predicate object map statement. Object Reference Map (ORM) implements the object reference between two triples maps defined over the same data source. It extends SOM by using the subject of the parent triples map as the object of another TM. Object Join Map (OJM) implements an index join in executing a join condition between two TMs defined over two diferent data sources. OJM resorts to the corresponding PJTT to access the encoded values in the child map associated with the encoded values of the data source of the parent triples map. After executing a TM and generating triples, each operator will compare each triple to its corresponding PTT for duplicate removal. If they exist in the PTT, the triples are ignored. If they do not exist in the PTT, the triples are added to the PTT and the KG.
The Workshop on Knowledge Graph Construction hosted the KGCW 2023 Challenge; a series of test cases that covered a wide range of configurations that would afect the KG creation process like TMs with a large number of predicateObjectMaps, TMs with joins of varying levels of complexity, TMs with data sources with high duplicate rate, to name a few. From these test cases, we define the following research questions: RQ1) What is the impact of the data duplicate rates in the execution time of SDM-RDFizer? RQ2) What is the impact of the input data size in the execution time of SDM-RDFizer? RQ3) How do the types of a triples maps afect the SDM-RDFizer? RQ4) How does the amount of triples maps afect the SDM-RDFIZER? This section provides a comprehensive overview of the empirical study conducted in this work. It encompasses the benchmarks, metrics, engine, and the experimental environment utilized to evaluate the performance of the engines. Each experimental configuration is repeated five times, and the average time and memory usage are reported as the outcome. The results were analyzed to determine the strengths and weaknesses of SDM-RDFizer.
Benchmark: The experiments were performed over the KGCW 2023 Challenge3 dataset. The
KGCW 2023 Challenge dataset was developed for the purpose of evaluating the performance (e.g.,
execution time and memory usage) of existing state-of-the-art KG creation engines. Additionally,
this dataset seeks not only to determine the fastest creation pipeline but the most eficient
pipeline based on the task, in other words, if the engine is the best suited for the configuration
case. This dataset includes some test cases extracted from the GTFS-Madrid-Bench [
RML engine and Metrics: The engine used to perform the KGCW 2023 Challenge was SDMRDFizer v4.7.1.54. This version of SDM-RDFizer used implements all the data structures and physical operators defined in the previous section. Alongside SDM-RDFizer, a tool (called
Duplicate 0% Duplicate 25% Duplicate 50% Duplicate 75% Duplicate 100% 76.77 56.76 38.86 20.40 2.41 Empty 0% Empty 25% Empty 50% Empty 75% Empty 100% 74.42 65.87 55.38 44.41 32.66 challenge-tool5) was provided for the challenge that will measure the execution time and memory usage. Therefore, the metrics used for the experimental study are execution time and memory usage. The execution time is reported in seconds (sec), and the memory usage is measured in megabytes (MB). For both execution time and memory usage, lower is better. The experiments are performed using in an Intel(R) Xeon(R) equipped with a CPU E5-1630 v4 @ 3.70GHz, 64GB memory, and with the O.S. Ubuntu 18.04LTS. The scripts used with challenge-tool, and the results generated from the challenge are available on GitHub 6.
Table 1 presents the results of the duplicate rate test cases. The percentage represents how much of the data source is defined as duplicates. It can be seen in the table that with higher duplicate rates, the execution time is shorter, and the memory usage is lower. This reduction in memory usage and execution time can be attributed to the fact that fewer unique triples will be generated with higher duplicate rates. Therefore, answering research question RQ1. Additionally, the duplicate removal process is faster due to the PTT from SDM-RDFizer. Table 2 presents the results of the empty values test cases. The percentage represents how much of the data source is defined by empty values. It can be seen in the table that the higher amount of empty values, the shorter the execution time and lower memory usage. Similar to the previous test case, a higher percentage of empty values suggests fewer unique triples are generated. The Empty 100% test case does not generate any triples.
Further, the test cases Duplicate 100% and Empty 100 % consume very little memory. This can be attributed to the fact that when only the table is given as the source of the TM, SDM-RDFizer generates based on the TM so that only necessary data is pulled from the database, thus avoiding the repetition of records. In the case of these test cases, each one only has one unique record. Thus, these test cases require very little to be transformed, leading to minimal memory usage. Table 3 illustrates the results of the mapping test cases. The mapping test cases illustrate how the number of the TMs and the number of predicateObjectMaps impact the KG creation process. It can be observed in the table that the number of TMs impacts much more than the number of predicateObjectMaps. The higher number of TMs increases the execution time; this can be attributed to the fact that even though the data sources are smaller when there are more TMs, SDM-RDFizer has to go through each data source individually, requiring time. When there is
1TM 15POM 3TM 5POM 5TM 3POM 15TM 1POM 66.13 104.59 147.69 362.10 1M rows 1 col 1M rows 10 col 1M rows 20 col 1M rows 30 col only one TM, the data source is larger, but SDM-RDFizer only has to go through it once, and the KG will be generated at the end. Therefore, answering RQ4.
Table 4 presents the results of the properties test cases. The properties test cases wish to present
the impact of the number of predicateObjectMaps on the KG creation process. For that reason,
the number of records in the data source was fixed to 1,000,000 rows. So, it can be seen in the
table the higher number of properties, execution time, and memory usage are higher. This
increase in memory usage and execution time can be attributed to the number of columns of the
data source, which in turn increases the size of the data source and confirms what Chaves-Fraga
et al. [
Table 5 presents the results of the records test cases; they seek to illustrate the impact of the number of rows in KG creation, so the number of columns is set to 20 for every test case. Similar to the properties test cases, these test cases reflect that with a higher number of rows, the space required to store them is more significant, and it needs more time to go through them. The results presented for the records and properties test cases illustrate the impact the size of the data sources have in the KG creation process, therefore answering research question RQ2. The results in Table 6 display the performance of test cases extracted from the GTFS-MadridBench benchmark, which defines multiple sizes of data sources, denoted by the scale indicating the size of the outputted KG. For example, KG at Scale 10 is ten times larger than Scale 1. As expected, larger data sources lead to higher execution time and memory usage. Notably, the operation causing the most overhead is a self-join involving the largest table, ”SHAPES.” Among the cases with ”Heterogeneity” in their names, which represent diferent combinations of data sources in various formats, all cases at Scale 100 show similar execution times and memory usage. This can be attributed to the use of XML and JSON files, which ofer faster Execution Time (sec) dictionaries results in slower traversal. JSON files are uploaded as hash tables, and XML files are represented as trees, which are more eficient data structures. The ”Heterogeneity Tabular” case, which exclusively uses CSV files and relational database tables, exhibits the highest execution time. This reinforces the notion that diferent data formats significantly impact performance. However, it is important to acknowledge the limitation that the largest scale, Scale 1000, was unable to be executed due to the characteristics of the environment in which the test cases were run. In conclusion, the analysis reveals that the choice of data format significantly afects the execution time and memory usage of the KG creation process, with XML and JSON files demonstrating better performance than CSV files and relational databases. Finally, Table 7, levels of complexity. It can be observed from all the tables that all test cases present very similar execution times. All test cases took around two minutes to be executed, with the highest execution time being 146.57. The similarity between the results can be linked to the usage of PJTTs. The PJTT helps SDM-RDFizer to avoid the need to upload the parent data source of a join multiple times, and by keeping the result of the join in main memory, the values can be extracted as needed, which in turn makes the execution of joins RML TMs much faster. The execution of joins requires a diferent approach and more resources to be transformed. That is why using PJTTs is necessary when the SDM-RDFizer executes joins. Thus, answering RQ3.
The KGCW 2023 Challenge dataset aims to evaluate state-of-the-art engines and assess their strengths and weaknesses. In the case of SDM-RDFizer, it demonstrated relatively low execution time and memory usage in task-oriented test cases, such as duplicate removal, join execution, and avoiding the generation of empty values. This performance improvement in SDM-RDFizer can be attributed to the optimized data structures implemented in the engine, enhancing various aspects of the KG creation process. However, due to the lack of results from other engines for comparison, it is challenging to determine the significance of these findings definitively. Nevertheless, looking toward future versions of SDM-RDFizer, the authors plan to incorporate new techniques for handling data sources. These advancements aim to reduce memory usage further and improve overall execution time. SDM-RDFizer is poised to enhance its performance and continue its trajectory as a state-of-the-art solution by actively addressing these areas.
This work has been partially supported by the Federal Ministry for Economic Afairs and Energy of Germany (BMWK) in the project CoyPu (project number 01MK21007[A-L]). Leibniz Association partially funds Maria-Esther Vidal in the ”Leibniz Best Minds: Programme for Women Professors”, project TrustKG-Transforming Data in Trustable Insights with grant P99/2020.