Recently, the W3C Community Group on Knowledge Graph Construction created a suite of test cases for all RML modules developed in the Community Group to verify implementations' compliance with the new RML specifications. However, these RML test cases could not be tested because no existing RML Processor supports them. In this paper, we report on our process of testing the new RML test cases while at the same time implementing support for the new RML modules in a reference implementation, which we call 'BURP' (Basic and Unassuming RML Processor), to investigate the feasibility and possible mistakes of the new RML test cases and specifications. We found several problems in the RML modules, ranging from mismatches between the test cases and their specification and invalid SHACL shapes to edge cases not covered by the specifications. Through this work, we improve the quality of RML test cases and the coverage of their corresponding specifications to increase adoption and conformance GitHub: https://github.com/kg-construct/BURP Fifth International Workshop On Knowledge Graph Construction Co-located with the ESWC 2024, May 26-27 2024, ∗Corresponding author. †These authors contributed equally.
CEUR ceur-ws.org
The W3C Community Group on Knowledge Graph Construction developed a suite of test
cases to ensure implementations comply with the new RML specifications [
The problem is that no existing reference implementation for RML exists. This lack of a reference algorithm results in inconsistencies between test cases, specifications, and RML https://dylanvanassche.be (D. Van Assche); http://christophedebruyne.be/ (C. Debruyne)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Processors, which hinders the community’s adoption of the new specifications.
In this work, we introduce BURP (Basic and Unassuming RML Processor ) to address this problem by providing a reference implementation on which the community can rely to develop new RML specifications, verify resources such as test cases and SHACL shapes, and compare implementations. BURP’s development was driven by the need to comply with and sanity check test cases across specifications for the Knowledge Graph Construction Workshop Challenge. 1 BURP aims to become a reference implementation for all new RML modules. BURP has been designed to keep things simple: all data is processed in main memory using simple data structures. BURP does not rewrite mappings to optimize RDF generation. BURP’s development has been instrumental in identifying and rectifying inconsistencies within the RML modules. This paper is structured as follows: Section 2 introduces our reference implementation BURP. Section 3 discusses all encountered problems in the RML test cases. Section 4 concludes this paper and discusses future work.
In this section, we introduce our reference RML implementation, the Basic and Unassuming RML Processor (BURP), which we developed to investigate the feasibility and consistency of the new RML specifications. We developed BURP from scratch to ensure we encounter all possible problems when implementing the RML modules and avoid any influence (i.e., assumptions) from existing RML Processors when establishing a reference implementation with a naive RDF generation algorithm.
BURP is basic since it uses simple data structures and an arguably naive approach to generate
RDF, unassuming as no optimizations are applied when implementing the RML specifications.
BURP follows simple steps to generate RDF, similar to the R2RML [
BURP currently fully supports RML-Core, RML-CC, and RML-FNML. It also supports some of the functionality of RML-IO (RML Logical Sources are supported, and the RML Logical Target is under development).
In the future, we aim to support all RML modules. BURP is written in Java, relying upon Apache’s Jena Semantic Framework for managing the generated RDF, and is available on GitHub under the MIT license. Therefore, this implementation aims to serve as the community’s sandbox for developing new RML modules and verifying their resources, e.g., RML mappings, SHACL shapes, test cases, etc.
In this Section, we discuss (i) the RML test cases of the diferent RML modules and (ii) how we discovered problems, edge cases, and inconsistencies between the resources of each RML module.
RML-Core contains the core mechanism of transforming data into an RDF Knowledge Graph,
such as RML Triples Maps and Predicate Object Maps. These core mechanisms were inherited
mainly from R2RML [
Compatibility with other (Web) standards was lacking among test cases. RML test cases using JSONPath as their reference formulation to refer to data from the RML mappings did not follow the latest IETF JSONPath specification. 3 When the original RML specification was introduced, a standardized JSONPath specification did not exist. Thus, many libraries implemented JSONPath slightly diferently in terms of syntax and supported features. The development of BURP and CARML4, another RML implementation, both strived to support the new JSONPath specification and thus uncovered this incompatibility. This problem was solved in the GitHub repository of RML-Core by updating the test cases.
Ill-formed language tags must be removed by an RML Processor when generating RDF according to the RML-Core specification since each language must be well-formed for RDF Literals according to BCP47.5 Test cases were provided to validate that Processors must enforce well-formed language tags, thus removing ill-formed language tags. However, the ill-formed examples in the test cases were, in fact, well-formed. Therefore, the test case did not achieve its goal of validating the compliance of RML Processors with the specification regarding ill-formed language tags. We discovered this problem as BURP did not report an ill-formed language tag when processing these test cases, leading us to investigate the problem.
Error handling by test cases was not clearly described. Test cases provided empty output ifles to test whether an RML mapping should cause a Processor to fail, but an empty file may very well be a valid output (an empty graph). Some test cases provided “partial” output to assess an RML Processor’s capability to continue generating RDF in case of an error. Together with the community, we decided that RML Processors must return a non-zero exit code when errors are encountered, which was not considered before. Before this, existing RML Processors would not cause scripts to report errors to developers as they always returned a zero exit code, even if the RML mapping failed. BURP always reports a non-zero exit code in case of failure and a zero exit code on success. We are considering with the community to establish a list of non-zero exit 2rml-core#75 rml-core#83, rml-core#89 rml-core#93, rml-core#100, rml-core#101 rml-core#109 3https://ietf-wg-jsonpath.github.io/draft-ietf-jsonpath-base/draft-ietf-jsonpath-base.txt 4https://github.com/carml/carml 5https://www.rfc-editor.org/rfc/rfc5646 codes for RML Processors with a predefined meaning to communicate errors, e.g., validation or access errors properly. Such an approach would allow RML implementation to retain and generate partial results.
Incomplete coverage was quickly discovered as test cases and specifications did not properly cover certain edge cases. For example, selecting attributes in XML data, data type conversions, handling of multi-valued Term Maps, and RML’s Datatype Map did not have a test case. Implementing BURP as a reference implementation forced the community to reconsider their decisions to ensure all edge cases are covered in specifications, test cases, and SHACL shapes.
Inconsistencies among relational databases are assumed to be non-existent by relying
on the ANSI SQL 2008 standard, inspired by R2RML [
Base IRI configuration of the RDF output generated from an RML mapping is impossible. An RML and R2RML mapping may contain a base IRI for the mapping, which was wrongly assumed to be the same base IRI for the RDF output generated by the RML mapping. This assumption was even stronger through the RML test cases since the R2RML test cases6 inspired them. Unfortunately, the R2RML test cases used the same base IRI as the base for the generated RDF (to be provided as input to the R2RML processor according to the R2RML specification) and the base of the mapping. There is currently an ongoing discussion7 within the W3C Community Group to allow the specification of a base IRI independent of the base IRI for the mapping.
SHACL shapes validation of RML mappings contained small mistakes such as a misleading
description, wrong predicates, or wrong node kinds. Thanks to the community, we can solve
these SHACL shape problems to reduce the burden on other developers of RML processors.
RML-IO handles all access with data sources by introducing an RML Logical Target [
Relative file paths were not considered when deprecating string Literals in rml:source
by [
Datatype inference was defined by the R2RML and the original RML specification to extract datatypes from relational databases by [R2]RML Processors. Still, such approaches for other heterogeneous data sources were not specified in RML. RML Processor developers proposed diferent approaches to determine a value’s datatype. This caused RML Processors to generate diferent RDF using the same RML mapping, such as a ‘64bit integer’ compared to a ‘32bit integer’ when the original data source specified an ‘integer’ as a datatype. Datatype inference is still under discussion10 and is considered to be moved to RML-IO to have all diferent data sources in RML-IO instead of RML-Core since RML-Core test cases should only focus only on the core mechanisms of RML instead of data source access descriptions. This method reduces the number of test cases in RML-Core as they are replicated for every data source. This problem was discovered as BURP does not assume anything and strictly follows the provided specifications. The community is considering specifying the datatype inference and corresponding reference formulations in a document to avoid inconsistencies among RML Processors when interpreting reference formulations and performing datatype inference.11
Under-specified vocabulary Since BURP only implements what is defined by the RML specifications, we noticed that the RML-IO vocabulary was under-specified. For example, the interpretation of rml:encoding with compressed files can be interpreted in 2 ways 12: (i) is it the encoding of the compressed file, (ii) or the encoding of the file being compressed? Therefore, the vocabulary must be more specific to define which file the encoding applies to.
Compatibility with other (Web) standards was also problematic for RML-IO when JSONPath expressions were used. The test cases did not follow the latest IETF JSONPath specification, similar to RML-Core (Section 3.1).
3.3. RML-CC
RML-CC is the RML module that supports generating RDF Collections (rdf:List) and
Containers (rdf:Alt, rdf:Bag, and rdf:Seq). This module is completely new, and no Processor
exists at the time of writing to support the RML-CC module. However, this module drew
inspiration from two predecessors: (i) an extension of R2RML-F [
Incomplete coverage of test cases as some edge cases only emerged implementing the RML-CC specification in BURP. We discovered that the test cases did not include gathering the terms generated by a Referencing Object Map. We also realized that the test cases assumed diferent base IRIs to generate RDF compared to other RML modules. While this is not an issue, the lack of coherence renders the suite of test cases less comprehensive to developers implementing an RML Processor.
SHACL validation errors with RML mappings were introduced by small changes in the vocabulary as contributors changed the specification and the corresponding test cases of other RML modules but did not update the SHACL shapes. Therefore, BURP raised mapping validation errors on some RML-CC test cases. Problems in SHACL shapes were resolved in their respective repositories.
Compatibility with other (Web) standards, similar to RML-Core (Section 3.1) and RMLIO (Section 3.2), updating to the latest IETF JSONPath specification was only discovered after implementing RML-CC in BURP.
The RML-Star module [13] was developed to support generating RDF-Star [14] around the same time the new RML modules were proposed. It was initially conceived for the original RML specification and later adapted to the new RML modules. RML-Star is currently only supported by a single RML Processor (Morph-KGCstar [15]). RML-Star had supporting test cases that had already been verified during the implementation of Morph-KGC. However, we discovered some problems when we started working on BURP.
Relative paths String Literals were deprecated and removed in RML-IO (cf. Section 3.2). However, the existing RML-Star test cases were not adjusted, and string literals were still used for the relative file paths of the test cases’ input data. We solved this problem for all RML-Star 13rml-cc#43, rml-cc#46, rml-cc#47 test cases in their GitHub. repository14
Validating RDF-Star output is currently challenging as not all existing RDF libraries, such as RDFLib15, support RDF-Star. This situation is improving as the RDF community implements RDF-Star in existing RDF libraries.
RML-FNML module aligns RML with FnO data transformations [16] to allow data transformations to be performed on the data during the mapping into RDF. FNML was heavily revised in the new RML modules to handle edge cases, e.g., multiple reference values, arrays, and other use cases. Morph-KGC [15] recently implemented support for the new RML-FNML module and verified its compliance through the corresponding RML-FNML test cases. However, when implementing BURP, and by only focussing on the specification, several problems 16 emerged:
The deprecation of relative paths as string Literals (Section 3.2) also afected the RML-FNML test cases. We adapted the existing test cases17 to use the RelativePathSource class instead of string Literals for rml:source.
Inconsistencies between SHACL shapes, the specification, and test cases, such as incorrect SHACL shapes to validate the test case’s RML mapping, small mistakes in predicate names, or specifying the behavior of multi-value edge cases. These small problems were discovered during BURP’s development, reported to and resolved by the RML-FNML maintainers.
Through this work, we increased the quality, coverage, and correctness of the RML test cases, specifications, and SHACL shapes by implementing a reference implementation ’BURP’ (Basic and Unassuming RML Processor) of the new RML modules and checking the BURP’s compliance with the RML modules. This way, we discovered several problems, e.g., mismatches between specifications, test cases, vocabulary, and edge cases that were not covered.
Thanks to this work, developers of RML Processors will encounter fewer problems complying with the new RML specifications and achieve a more consistent execution behavior. This exercise has also informed us and the community about the need for further extension and introduction of test cases, depending on consensus within the community. In the future, we aim to extend BURP further together with the community to cover all RML modules, demonstrating the community’s integral role in the ongoing development process.
Dylan Van Assche is supported by the Special Research Fund of Ghent University18 under grant BOF20/DOC/132. The collaboration of Dylan Van Assche and Christophe Debruyne is 14rml-star#27 15https://github.com/RDFLib/RDFlib 16rml-fnml#33, rml-fnml#34 17rml-fnml#35 18https://www.ugent.be/en/research/funding/bof/overview.htm stimulated by the KG4DI FWO scientific research network (W001222N). [11] C. Debruyne, L. McKenna, D. O’Sullivan, Extending R2RML with support for RDF collections and containers to generate MADS-RDF datasets, in: Research and Advanced Technology for Digital Libraries - 21st International Conference on Theory and Practice of Digital Libraries, TPDL 2017, Thessaloniki, Greece, September 18-21, 2017, Proceedings, volume 10450 of Lecture Notes in Computer Science, Springer, 2017, pp. 531–536. [12] F. Michel, L. Djimenou, C. Faron-Zucker, J. Montagnat, Translation of Relational and Non-relational Databases into RDF with xR2RML, in: WEBIST 2015 - Proceedings of the 11th International Conference on Web Information Systems and Technologies, Lisbon, Portugal, 20-22 May, 2015, SciTePress, 2015, pp. 443–454. [13] J. Arenas-Guerrero, A. Iglesias-Molina, D. Chaves-Fraga, D. Garijo, O. Corcho, A. Dimou, Declarative generation of RDF-star graphs from heterogeneous data, Submitted to Semantic Web (2024). [14] H. Olaf, C. Pierre-Antoine, K. Gregg, S. Andy, RDF-star and SPARQL-star, Final Community Group Report, World Wide Web Consortium (W3C), 2021. URL: https://w3c.github.io/ rdf-star/cg-spec. [15] J. Arenas-Guerrero, D. Chaves-Fraga, J. Toledo, M. S. Pérez, O. Corcho, Morph-KGC: Scalable knowledge graph materialization with mapping partitions, Semantic Web 15 (2024) 1–20. [16] B. De Meester, T. Seymoens, A. Dimou, R. Verborgh, Implementation-independent Function Reuse, Future Generation Computer Systems 110 (2020) 946–959.