Data mapping languages allow users to create knowledge graphs with lower cost and time. Some challenges cannot be solved with state-of-the-art languages and tools, though. Thus, in this paper we use and modify ShExML to deal with some of them. We see how some challenges were already solved, which modifications we had to perform to cover others, and how the rest of them could be covered in future versions. Then, we establish a demonstration on language integrity after changes and a discussion on performed and upcoming changes. These solutions, alongside the discussion and joint analysis of other languages and tools solutions, will drive us to effective techniques to solve all proposed challenges.
Mapping heterogeneous datasources using a single representation is an active field which has been getting traction in the past years. For this purpose a set of languages and tools has been proposed [8] which lower the cost and time employed in these tasks. This trajectory ended up in the celebration of the 1st International Workshop on Knowledge Graph Building [1] and, one year after, the beginning of the Knowledge Graph Construction W3C Community Group 1 where academia, industry and practitioners are gathered to envision new steps, find unsolved problems, and face new challenges in this field 2 . One of the community outputs was a set of mapping challenges 3 which are nowadays complicated to solve with the current state-of-the-art languages, tools and techniques. Therefore, in this paper we tackle some of these mapping challenges with ShExML [5] and try to solve the following questions:
-Q1: How can mapping challenges be solved with ShExML? -Q2: How can unaddressed challenges be solved and implemented in ShExML? -Q3: Have modifications in ShExML affected the functioning of already present features?
The rest of the paper is structured as follows: In Section 2 we summarise the mapping challenges proposed by the community; additionally, we offer a set of suplementary material to support following explanations. In Section 3 we see how the current language specification and engine can solve some of the challenges. Then, we explain how the solutions for other challenges have been implemented and included in ShExML in Section 4. In Section 5 we propose some further language modifications and a discussion on how the rest of the challenges could be addressed. We demonstrate the old features integrity after including the new ones and we establish a discussion on mapping challenges results in Section 6. And, finally, in Section 7 we draw some conclusions.
During consecutive meetings in the Knowledge Graph Construction W3C Community Group, several mapping challenges and problems were arisen which are collected in the workshop website 4 . Thus, in this paper we deal with this selection of mapping challenges, we examine them and propose solutions within the ShExML language and our engine5 . To categorise these solutions we link them to ShExML versions so we can trace when these solutions were achieved, i.e., if they were possible to solve before the mapping challenges were defined (ShExML v0.2.3), if they were solved after the mapping challenges were defined (ShExML v0.2.4 & v0.2.5) or if they are not yet solved (future versions).
Thus, in Table 1 a summary table is offered with the addressed challenges and with which version of ShExML engine the expected output is achieved. Besides, we offer a webpage6 with links to the working solutions as supplementary material for the sake of demonstration and reproducibility. The inputs are taken from the Knowledge Graph Construction W3C Community Group mapping challenges repository 7 which contains a set of inputs and expected outputs that are agreed to represent the community mapping problems.
In the following sections we explain how solutions are achieved, which ShExML constructions and techniques were used, we establish a discussion on reached solutions and how unsolved challenges could be addressed in ShExML.
In this section, we deal with the mapping challenges that can be solved without any modification in the ShExML language and engine. Therefore, these solutions are those that are reachable with ShExML v0.2.3 (released on 29th October 2020) 8 , that is, before the mapping challenges were defined.
Datatype map refers to the possibility to generate datatype tags from the input content. Therefore, instead of defining them statically in the mapping rules, this challenge aims to support the dynamic generation of datatype tags from input content. In the case of input 5, it is intended that mapping languages would be able to generate datatype tags according to the most probable value according to values formats. For example, in input 5 it is expected that the number 3 would have an xsd:integer datatype and 3.14 would have an xsd:decimal one. This inference mechanism was already implemented in ShExML engine which in case that the user does not specifically define a datatype for an object value it will infer the most probable one (see Listing 1.1). Although the current implementation solves this specific mapping challenge, it is a naïve implementation.
However, it can lead to a more complex inference system, for example, aligning existing input data sources datatypes with RDF ones.
This challenge refers to the possibility to generate literals from a join condition (i.e., from another source) where R2RML 9 and RML [3] output a resource by default.
In ShExML, join conditions10 generate values without any specific form, so it is not determined in this step if it is a literal or a resource. It is, then, defined by the user in the shapes part, where the user decides the form of the output. This is a design decision on ShExML that was driven by the separation of concerns main principle. In Listing 1.2 we can see how the join condition is defined in familyName expression, and how then this expression is used in :Author shape without any prefix, indicating that a literal must be generated.
This challenge deals with the expected output of a hierarchical document (e.g., XML or JSON files) where multiple iterators are used. The discussion in this challenge is whether we produce the cartesian product and provide a join condition to correlate values or if we just translate the hierarchical information as it is, without the need to provide any join condition 11 . This case becomes more complicated if a join condition needs to be provided over a JSON file because of the impossibility to access parent nodes (see Section 4.1 for the specific challenge on this topic). Therefore, it seems that in hierarchical data the expected output should be a verbatim translation (so to say, not creating the cartesian product as it is not how it is originally represented in the input file).
In ShExML, this was the default behaviour from its inception as in ShExML first versions it only supported XML and JSON files. Besides, we saw it as a more usable manner to define these mappings as usability is the main goal of the language [4]. Therefore, in Listing 1.3 we can see how using iterators and nested iterators we can cover these hierarchical data models. In Table 1 this challenge is marked as not solved in ShExML v0.2.3 due to a bug when using only the root node ($) in the top iterator query. However, we include it here as the coverage of this challenge did not require syntax modifications nor new features in ShExML engine, only a bug fix.
In this section, we deal with the mapping challenges that needed some modifications in the ShExML language and engine. Therefore, these solutions are those which are reachable with ShExML v0.2.4 (released on 18th January 2021) 12or with ShExML v0.2.5 (released on 27th January 2021) 13 , that is, after the mapping challenges were defined.
Sometimes, in hierarchical data models, there is the need to access values outside the iteration pattern. For example, we may need to obtain the values that are parents of the current iterated node. When dealing with XML files it does not involve any modification in ShExML, as using XPath queries we are able to access upper nodes with the double dot and slash notation (i.e., ../). However, when dealing with JSON files, this is not possible because of JSONPath not supporting the parent access notation 14 . This is a well-known problem in data mapping languages as they use Json-Path to define values accesses. Indeed, in xR2RML [11], the authors defined a property called xrr:pushDown that takes a value in the hierarchy and pushes it down into their offsprings iterators so it can be available further [12].
Following this experience with xR2RML, we implemented a similar solution in ShExML using the PUSHED_FIELD and POPPED_FIELD keywords. When using the PUSHED_FIELD keyword the ShExML engine saves the value using the name as the identifier for further uses. Then, when the POPPED_FIELD is used the ShExML engine searches for the saved value with an identifier which is equal to that given in the query part (i.e., inside < and >). Therefore, in Listing 1.4, the id field is saved and then used in the cars iterator, so we can establish a relation from the car to the owner.
An interesting discussion here is if it is better to make these accesses implicit or explicit. A recent RML syntax modification proposal [2] presented an algorithm that could implicitly access values from upper hierarchical levels by saving iteration information, indexes and values. Users can directly access upper elements from its current hierarchical level. Indeed, the solution is clean and avoids the user's explicit declaration of values to be saved. However, it has two possible main drawbacks. Firstly, the use of a bigger amount of memory and time by saving a lot information that could be used (or not) in further mapping rules. This technique, could be, in the end, a bottleneck in performance if it is not carefully implemented. Second one, it could complicate the engine implementation as it allows to go up and down in the hierarchy, while actual behaviour only expects to go down. This could also lead to a performance issue. Either way, this dichotomy should be quantified in further experiments to establish the best solution in terms of usability and performance.
As we mentioned in Section 3.1, this challenge aims to generate datatype tags dynamically from data content. Therefore, the datatype inputs can appear in multiple ways: full URI, prefix plus datatype, or simply datatype name without prefix.
ShExML v0.2.3 supports the creation of static datatype tags with prefix plus datatype syntax (see Listing 1.5). Therefore, we should derive this syntax and maintain its proven usability [4] but giving dynamic datatype generation possibilities. The natural expansion of this syntax is to include the same object generation expression but also for datatypes and language tags (see Section 4.3). So, the final syntax is prefix plus generation expression (inside square brackets) as we can see in Listing 1.6. The prefix can be optional if the data value already contains it (e.g., input 1 and 2) and values can be transformed using Matcher feature15 to expected XML Schema valid datatypes (e.g., input 4).
As with datatype maps in Section 4.2, the language map challenge want to address the problem of generating language tags dynamically from input data. In ShExML v0.2.3 language tags were supported statically, that is, it was possible to tag an object expression with a specific language but it would be applied to all values (see Listing 1.7).
We performed a syntax and engine modification, like in datatype maps, to be able to generate language tags with expressions. The final syntax is @ plus the generation expression (between square brackets) as it can be seen in Listing 1.8. Here, again, the idea was to preserve the usability as the main goal and to make it as simple as possible. Input 1 tests the generation with a valid tag following BCP4716 , input 2 tests the transformation of a language value to a valid tag (in ShExML this is done using Matchers functionality17 ), and in input 3 it is shown how two different sources can be joined to provide language information.
As a side note, it is not possible to specify a language map and a datatype map in the same triple, as it is forbidden by the ShExML grammar. This was made intentional to follow the RDF specification rules as it can be seen in its grammar 18 . Therefore, whenever a langtag generation expression (either static or dynamic) is provided the implicit datatype is rdf:langString.
This challenge wants to address the problem of generating various datatypes or language tags for the same subject (e.g., a multi-language value). Once datatype maps (see Section 4.2) and language maps (see Section 4.3) are supported in ShExML, it is straightforward as ShExML will generate a triple per value returned from the object expression. Therefore, to generate multi-language values the syntax is like in Listing 1.9 and to generate multi-language values with a default language the syntax is like in Listing 1.10.
This challenge puts on the table the necessity for a mechanism to create RDF Collections from some values. Normally, in ShExML, and in other data mapping languages, when an object generation expression returns multiple values, multiple triples are generated (see Section 3.3). However, in certain cases it is necessary to encapsulate these values inside a collection (e.g., to preserve order).
This was already explored by some languages (e.g., SPARQL-Generate [7] and xR2RML [11]) which provide some directives to create collections. Therefore, we applied this experience in ShExML to cover RDF Collections and Containers (i.e., Lists, Seqs, Bags and Alts.). Now, it is possible to indicate to the engine that a collection or container should be generated using the keyword AS plus the desired collection or container (i.e., RDFList, RDFBag, RDFSeq or RDFAlt). The proposed syntax follows the same design principles from already existing features syntax (e.g., Matchers19 ). See Listing 1.11 for an example. In this section we discuss further challenges that are not solved with the previously mentioned modifications. These are the challenges that would require to rethink some functionality, or to include new ones, but that would need from a well planned inclusion, due to their possible interference with other features.
Although this challenge was already addressed in Section 4.1, only input 1 was completely solved. In the case of input 2, where data is in the same hierarchical level (like it would come from two different files), using join conditions in ShExML, only one car is linked to each owner when the expected result was two cars per person. To solve this problem we think of two possible solutions.
First one is to review the join condition functionality to check whether something is failing (a bug) or if the join condition need to be rethought and reimplemented to cover further challenges.
Another possibility, which is already present in other languages like YARRRML [6], is to provide conditional generation. With conditional content generation we are able to test a condition (e.g., in input 2 for value equality) and generate or not the resulting triple depending on its result.
A classic solution when dealing with Excel sheets was to convert them to CSV, and then treat them as tables to be processed by data mapping languages. However, this challenge found this solution not appropriate when the style of the Excel sheet is wanted to be preserved. Two solutions could cover this challenge.
First one is to preprocess the Excel sheet and convert it to CSV but adding columns with style information so it can be processed by state-of-the-art tools. However, it would require some preprocessing work which would weaken the goal of low cost and time invested when using data mapping tools.
Second one is to include a specific Excel processor, with its own query language, which can express not only access to cells, but also to the cell and text style. Thus, in Java based implementations it can be considered to use Apache POI to process sheets, and include some simple query support to retrieve styles. Therefore, in ShExML this would require to support the mentioned query language for Excel, and then, integrate it into the ShExML engine to retrieve Excel sheets values.
This challenge is very close to multivalue references (see Section 3.3), but in this case multivalues are included all within a string value and separated by commas.
Therefore, here the challenge is not about how to output multivalues, or create RDF collections, but how to effectively handle these multivalues which need some processing. Therefore, this would require some sort of data transformations functions that can be applied to the extracted values. Therefore, the most effective way to extend ShExML and enable users to transform data is to provide the possibility to execute transformation functions which can be defined by users.
Data transformation functions have been already explored in RML through the FnO library [9] which provides a set of implementation independent reusable functions [10]. So, one possibility is to support FnO functions inside ShExML. The advantage of this proposal is that it moves all function infrastructure outside the ShExML language and engine. Conversely, we add more dependencies to users (which can find it hard to learn), we force them to use a third party environment and we lose control of this part.
Another possibility is to provide an environment to define inline functions like the semantic actions in Shape Expressions (ShEx) [13]. Therefore, we can provide a restricted environment where higher order functions could be executed (see Listing 1.12 for an example). The advantages are that there is no need for third party dependencies, it provides a higher flexibility and users do not need to learn another tool. However, it can increase complexity due to the necessity to know about functional programming.
Although RDF collections and containers were included in ShExML (see Section 4.5), input 2 and 3 present some particularities. In the case of input 2 the use of different keys would require a more complex query or some sort of parametrisation in executed queries. In input 3 the per row iteration model for CSV files implemented in ShExML does not create collections effectively. Therefore, it would imply a reimplementation of per row iteration model for these cases. However, it could affect the overall functionality for CSV files.
In Q3 we have posed a question about the possible effects that the modifications in ShExML could have in already working features. The idea of this question is to demonstrate the validity of Q1 solutions alongside old features that should still work as expected. This type of testing, known as regression testing, have been included in ShExML from the very beginning20 so we are able to add new features in ShExML knowing that old features are still working as expected. Thus, every time a new version is released these tests must be executed to validate the language and engine integrity. Continuous integration is the perfect tool for this task, as every time that a change is submitted to ShExML repository all tests are executed to verify the integrity. In the ShExML repository we have configured Travis CI21 for this task. Therefore, these regression tests in v0.2.422 and v0.2.523 are telling us that all features are still working as expected, and equally, giving a negative answer to Q3. So, we can conclude that integrity is held.
In Sections 3 and 4 we have seen how some mapping challenges were already solved in ShExML and how we have made some modifications in ShExML language and engine to deal with the others. These two sections give an answer for Q1. These solutions were designed to maintain ShExML usability [4] using a similar syntax, so that users can use these new features with the minimum learning curve possible; in other words, making the smallest modifications in the ShExML syntax. In addition, in Section 3 we have highlighted how the ShExML design have already given an answer to some challenges, emphasising how the ShExML separation of concerns principle can give answers to some of them (e.g., Join on literal).
In Section 5 we have given some intuition on how remaining challenges could be solved, answering to Q2. They would require harder and more complex modifications; in some cases the modification of an already working mechanism (e.g., inputs 2 and 3 in RDF Collections), the inclusion of a new iteration model and the design of a new query language (e.g., Excel style) or the choice between two different systems (e.g., data transformation functions in process multivalue references). All these inclusions will require a careful study and implementation in the language so they do not affect other features and, also, to select the better option from a usability perspective.
In this paper we have explored how ShExML can deal with some of the challenges defined in the Knowledge Graph Construction W3C Community Group. We have divided them into challenges already solved by ShExML before their definition, challenges solved by latest versions of ShExML, and challenges that are not yet solved, for which we have given some notions and intuitions on how ShExML can be modified to cover them. Furthermore, we have demonstrated that the modification of ShExML to cover new challenges has not affected other language and engine features. Therefore, we see this work as a first step on demonstrating how the challenges can be solved and, together with solutions from other languages and the joint discussion, we will be able to offer unified solutions to the posed mapping challenges.