Data mapping languages allow users to create knowledge graphs with lower cost and time. Some challenges cannot be solved with state-of-the-art languages and tools, though. Thus, in this paper we use and modify ShExML to deal with some of them. We see how some challenges were already solved, which modi cations we had to perform to cover others, and how the rest of them could be covered in future versions. Then, we establish a demonstration on language integrity after changes and a discussion on performed and upcoming changes. These solutions, alongside the discussion and joint analysis of other languages and tools solutions, will drive us to e ective techniques to solve all proposed challenges.
Mapping heterogeneous datasources using a single representation is an active
eld which has been getting traction in the past years. For this purpose a set
of languages and tools has been proposed [
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 https://w3id.org/kg-construct/tpac/ 2 https://w3id.org/kg-construct/tpac/#report 3 https://w3id.org/kg-construct/workshop/challenges.html
Therefore, in this paper we tackle some of these mapping challenges with
ShExML [
The rest of the paper is structured as follows: In Section 2 we summarise the mapping challenges proposed by the community; additionally, we o er a set of suplementary material to support following explanations. In Section 3 we see how the current language speci cation and engine can solve some of the challenges. Then, we explain how the solutions for other challenges have been implemented and included in ShExML in Section 4. In Section 5 we propose some further language modi cations and a discussion on how the rest of the challenges could be addressed. We demonstrate the old features integrity after including the new ones and we establish a discussion on mapping challenges results in Section 6. And, nally, in Section 7 we draw some conclusions. 2
During consecutive meetings in the Knowledge Graph Construction W3C Community Group, several mapping challenges and problems were arisen which are collected in the workshop website4. Thus, in this paper we deal with this selection of mapping challenges, we examine them and propose solutions within the ShExML language and our engine5. To categorise these solutions we link them to ShExML versions so we can trace when these solutions were achieved, i.e., if they were possible to solve before the mapping challenges were de ned (ShExML v0.2.3), if they were solved after the mapping challenges were de ned (ShExML v0.2.4 & v0.2.5) or if they are not yet solved (future versions).
Thus, in Table 1 a summary table is o ered with the addressed challenges and with which version of ShExML engine the expected output is achieved. Besides, we o er a webpage6 with links to the working solutions as supplementary material for the sake of demonstration and reproducibility. The inputs are taken from the Knowledge Graph Construction W3C Community Group mapping challenges repository7 which contains a set of inputs and expected outputs that are agreed to represent the community mapping problems.
In the following sections we explain how solutions are achieved, which ShExML constructions and techniques were used, we establish a discussion on reached solutions and how unsolved challenges could be addressed in ShExML.
Already solved With language modi cations (v0.2.3) (v0.2.4 & v0.2.5) x X x x x X x X x X x X X X
In this section, we deal with the mapping challenges that can be solved without any modi cation in the ShExML language and engine. Therefore, these solutions are those that are reachable with ShExML v0.2.3 (released on 29th October 2020)8, that is, before the mapping challenges were de ned. 3.1
Datatype map refers to the possibility to generate datatype tags from the input content. Therefore, instead of de ning them statically in the mapping rules, this challenge aims to support the dynamic generation of datatype tags from input content. In the case of input 5, it is intended that mapping languages would be able to generate datatype tags according to the most probable value according to values formats. For example, in input 5 it is expected that the number 3 would have an xsd:integer datatype and 3.14 would have an xsd:decimal one.
This inference mechanism was already implemented in ShExML engine which in case that the user does not speci cally de ne a datatype for an object value it will infer the most probable one (see Listing 1.1). Although the current implementation solves this speci c mapping challenge, it is a nave implementation. 8 https://github.com/herminiogg/ShExML/releases/tag/v0.2.3 However, it can lead to a more complex inference system, for example, aligning existing input data sources datatypes with RDF ones.
Listing 1.1. ShExML datatype inference function. protected def searchForXSDType (o: String ): RDFDatatype = { if( Try (o. toInt ). isSuccess )
XSDDatatype . XSDinteger else if( Try (o. toDouble ). isSuccess )
XSDDatatype . XSDdecimal else if( Try (o. toBoolean ). isSuccess )
XSDDatatype . XSDboolean else
XSDDatatype . XSDstring }
This challenge refers to the possibility to generate literals from a join condition
(i.e., from another source) where R2RML9 and RML [
In ShExML, join conditions10 generate values without any speci c form, so it is not determined in this step if it is a literal or a resource. It is, then, de ned by the user in the shapes part, where the user decides the form of the output. This is a design decision on ShExML that was driven by the separation of concerns main principle. In Listing 1.2 we can see how the join condition is de ned in familyName expression, and how then this expression is used in :Author shape without any pre x, indicating that a literal must be generated.
Listing 1.2. ShExML solution for join on literals.
PREFIX : <http: // example . com /> PREFIX experson: <http: // example . com / person /> PREFIX dbr: <http: // dbpedia . org / resource /> PREFIX schema: <http: // schema . org /> SOURCE jsonfile <https: // raw . githubusercontent . com / kg - construct / mapping - challenges / 2 aac9680cd731fd647abd33d44a7f400e4278cf3 / challenges /join -on - literal /input -1/ input .json > ITERATOR author < jsonpath: $. author [*] > {
FIELD id <id > FIELD firstname <firstname >
FIELD affiliation <affiliation > } ITERATOR people < jsonpath: $. people [*] > {
10 See ShExML speci cation for a full explanation on how join construction works: http://shexml.herminiogarcia.com/spec/#join FIELD firstname <firstname >
FIELD familyname <familyName > } EXPRESSION authors < jsonfile . author UNION jsonfile . people > EXPRESSION familyName < jsonfile . people . familyname UNION jsonfile . author . firstname JOIN jsonfile . people . firstname > :Author experson: [ authors .id] { :affiliation [ authors . affiliation ] ; :lastName [ familyName ] ; }
This challenge deals with the expected output of a hierarchical document (e.g., XML or JSON les) where multiple iterators are used. The discussion in this challenge is whether we produce the cartesian product and provide a join condition to correlate values or if we just translate the hierarchical information as it is, without the need to provide any join condition11. This case becomes more complicated if a join condition needs to be provided over a JSON le because of the impossibility to access parent nodes (see Section 4.1 for the speci c challenge on this topic). Therefore, it seems that in hierarchical data the expected output should be a verbatim translation (so to say, not creating the cartesian product as it is not how it is originally represented in the input le).
In ShExML, this was the default behaviour from its inception as in ShExML
rst versions it only supported XML and JSON les. Besides, we saw it as a
more usable manner to de ne these mappings as usability is the main goal of
the language [
Listing 1.3. ShExML solution multivalue references.
PREFIX ex: <http: // example . com /> PREFIX exLab: <http: // example . com / lab /> PREFIX exArticle: <http: // example . com / article /> PREFIX exAuthor: <http: // example . com / author /> PREFIX exAff: <http: // example . com / aff /> SOURCE lab_file <https: // raw . githubusercontent . com / kg - construct / mapping - challenges / main / challenges / multivalue - references / input -1/ input .json > 11 See discussion about this topic in RMLMapper reference implementation: https: //github.com/RMLio/rmlmapper-java/issues/28 ITERATOR lab < jsonpath: $> {
FIELD labName <labName > ITERATOR articles < articles [*] > {
FIELD title <title > ITERATOR authors <authors [*] > {
FIELD name <name > ITERATOR affiliation < affiliation [*] > {
FIELD label <label > EXPRESSION labValues < lab_file .lab > ex:Lab exLab: [ labValues . labName ] { a ex:Lab ; ex:hasArticles @ ex:Article ; ex:hasMembers @ ex:Author ; ex:Article exArticle: [ labValues . articles . title ] {
ex:hasAuthor @ ex:Author ; ex:Author exAuthor: [ labValues . articles . authors . name ] { ex:hasAffiliation
exAff: [ labValues . articles . authors . affiliation . label ] ; } } }
} } } }
In this section, we deal with the mapping challenges that needed some modi cations in the ShExML language and engine. Therefore, these solutions are those which are reachable with ShExML v0.2.4 (released on 18th January 2021)12 or with ShExML v0.2.5 (released on 27th January 2021)13, that is, after the mapping challenges were de ned.
Sometimes, in hierarchical data models, there is the need to access values outside the iteration pattern. For example, we may need to obtain the values that are parents of the current iterated node. When dealing with XML les it does not involve any modi cation in ShExML, as using XPath queries we are able to 12 https://github.com/herminiogg/ShExML/releases/tag/v0.2.4 13 https://github.com/herminiogg/ShExML/releases/tag/v0.2.5 access upper nodes with the double dot and slash notation (i.e., ../). However, when dealing with JSON les, this is not possible because of JSONPath not supporting the parent access notation14.
This is a well-known problem in data mapping languages as they use
JsonPath to de ne values accesses. Indeed, in xR2RML [
Following this experience with xR2RML, we implemented a similar solution in ShExML using the PUSHED_FIELD and POPPED_FIELD keywords. When using the PUSHED_FIELD keyword the ShExML engine saves the value using the name as the identi er for further uses. Then, when the POPPED_FIELD is used the ShExML engine searches for the saved value with an identi er which is equal to that given in the query part (i.e., inside < and >). Therefore, in Listing 1.4, the id eld is saved and then used in the cars iterator, so we can establish a relation from the car to the owner.
An interesting discussion here is if it is better to make these accesses implicit
or explicit. A recent RML syntax modi cation proposal [
Listing 1.4. ShExML solution for accessing elds outside iterations. ITERATOR records < j so np at h: $ . records [*] > {
P U S H E D _ F I E L D id <id > FIELD en te re dB y < enteredBy > ITERATOR cars < cars [*] > {
FIELD make < make >
P O P P E D _ F I E L D carOwner <id > }
} 4.2
As we mentioned in Section 3.1, this challenge aims to generate datatype tags dynamically from data content. Therefore, the datatype inputs can appear in 14 https://goessner.net/articles/JsonPath/ multiple ways: full URI, pre x plus datatype, or simply datatype name without pre x.
ShExML v0.2.3 supports the creation of static datatype tags with pre x
plus datatype syntax (see Listing 1.5). Therefore, we should derive this syntax
and maintain its proven usability [
Listing 1.5. ShExML static datatypes syntax. ex:Person exPerson: [ person . firstname ] {
ex:num [ person . num ] x s d : i n t e g e r ; } }
Listing 1.6. ShExML dynamic datatypes syntax. ex:Person exPerson: [ person . firstname ] {
ex:num [ person . num ] xsd: [ person . dt ] ; 4.3
As with datatype maps in Section 4.2, the language map challenge want to address the problem of generating language tags dynamically from input data. In ShExML v0.2.3 language tags were supported statically, that is, it was possible to tag an object expression with a speci c language but it would be applied to all values (see Listing 1.7).
We performed a syntax and engine modi cation, like in datatype maps, to be able to generate language tags with expressions. The nal syntax is @ plus the generation expression (between square brackets) as it can be seen in Listing 1.8. Here, again, the idea was to preserve the usability as the main goal and to make it as simple as possible. Input 1 tests the generation with a valid tag following BCP4716, input 2 tests the transformation of a language value to a valid tag (in ShExML this is done using Matchers functionality17), and in input 3 it is shown how two di erent sources can be joined to provide language information.
As a side note, it is not possible to specify a language map and a datatype map in the same triple, as it is forbidden by the ShExML grammar. This was made intentional to follow the RDF speci cation rules as it can be seen in its 15 http://shexml.herminiogarcia.com/spec/#matcher 16 https://tools.ietf.org/html/bcp47 17 http://shexml.herminiogarcia.com/spec/#matcher grammar18. Therefore, whenever a langtag generation expression (either static or dynamic) is provided the implicit datatype is rdf:langString.
Listing 1.7. ShExML static generation of language tags. ex:Person exPerson: [ person . firstname ] {
ex:lastName [ person . lastname ] @ en ; } } } } 4.4
This challenge wants to address the problem of generating various datatypes or language tags for the same subject (e.g., a multi-language value). Once datatype maps (see Section 4.2) and language maps (see Section 4.3) are supported in ShExML, it is straightforward as ShExML will generate a triple per value returned from the object expression. Therefore, to generate multi-language values the syntax is like in Listing 1.9 and to generate multi-language values with a default language the syntax is like in Listing 1.10.
Listing 1.9. ShExML multiple values with language tags. ex:Person exPerson: [ person . lastname ] {
ex:name [ person . firstname . label ] @[ person . firstname . lang ] ; Listing 1.10. ShExML multiple values with language tags and with a default language. ex:Person exPerson: [ person . firstname ] { ex:name [ person . firstname ] @ en ; ex:name [ person . firstname ] @[ person . lang ] ;
Listing 1.8. ShExML dynamic generation of language tags ex:Person exPerson: [ person . firstname ] {
ex:lastName [ person . lastname ] @[ person . lang ] ; 4.5
This challenge puts on the table the necessity for a mechanism to create RDF Collections from some values. Normally, in ShExML, and in other data mapping languages, when an object generation expression returns multiple values, multiple triples are generated (see Section 3.3). However, in certain cases it is necessary to encapsulate these values inside a collection (e.g., to preserve order). 18 https://www.w3.org/TR/turtle/#h3 sec-grammar-grammar
This was already explored by some languages (e.g., SPARQL-Generate [
Listing 1.11. ShExML support for RDF collections and containers. e x : A r t i c l e e x A r t i c l e : [ la bV al ue s . articles . title ] { a e x : A r t i c l e ; e x : h a s A u t h o r s
e xA ut hor : [ la bV al ue s . articles . authors . name AS RDFList ] ; } 5
In this section we discuss further challenges that are not solved with the previously mentioned modi cations. These are the challenges that would require to rethink some functionality, or to include new ones, but that would need from a well planned inclusion, due to their possible interference with other features. 5.1
Although this challenge was already addressed in Section 4.1, only input 1 was completely solved. In the case of input 2, where data is in the same hierarchical level (like it would come from two di erent les), using join conditions in ShExML, only one car is linked to each owner when the expected result was two cars per person. To solve this problem we think of two possible solutions.
First one is to review the join condition functionality to check whether something is failing (a bug) or if the join condition need to be rethought and reimplemented to cover further challenges.
Another possibility, which is already present in other languages like YARRRML
[
A classic solution when dealing with Excel sheets was to convert them to CSV, and then treat them as tables to be processed by data mapping languages. However, this challenge found this solution not appropriate when the style of the Excel sheet is wanted to be preserved. Two solutions could cover this challenge. 19 http://shexml.herminiogarcia.com/spec/#matcher-0
First one is to preprocess the Excel sheet and convert it to CSV but adding columns with style information so it can be processed by state-of-the-art tools. However, it would require some preprocessing work which would weaken the goal of low cost and time invested when using data mapping tools.
Second one is to include a speci c Excel processor, with its own query language, which can express not only access to cells, but also to the cell and text style. Thus, in Java based implementations it can be considered to use Apache POI to process sheets, and include some simple query support to retrieve styles. Therefore, in ShExML this would require to support the mentioned query language for Excel, and then, integrate it into the ShExML engine to retrieve Excel sheets values. 5.3
This challenge is very close to multivalue references (see Section 3.3), but in this case multivalues are included all within a string value and separated by commas.
Therefore, here the challenge is not about how to output multivalues, or create RDF collections, but how to e ectively handle these multivalues which need some processing. Therefore, this would require some sort of data transformations functions that can be applied to the extracted values. Therefore, the most e ective way to extend ShExML and enable users to transform data is to provide the possibility to execute transformation functions which can be de ned by users.
Data transformation functions have been already explored in RML through
the FnO library [
Another possibility is to provide an environment to de ne inline functions
like the semantic actions in Shape Expressions (ShEx) [
Listing 1.12. ShExML support for RDF collections and containers.
PREFIX ex: < http: // example . com /> SOURCE lab_file < https: // raw . githubusercontent . com / kg - construct / mapping - challenges / main / challenges / process - multivalue - references / input -1/ input . json > FUNCTION splitFunction <n => n. split ( ',')> ITERATOR lab < jsonpath: $ > {
FIELD labName < labName > ITERATOR articles < article > { }
FIELD title < title >
FIELD tags < tags > } } E X P R E S S I O N la bV al ue s < lab_file . lab > ex:Tag ex: [ lab . articles . tag WITH s p l i t F u n c t i o n ] {
ex:label [ lab . articles . tag WITH s p l i t F u n c t i o n ] ; Although RDF collections and containers were included in ShExML (see Section 4.5), input 2 and 3 present some particularities. In the case of input 2 the use of di erent keys would require a more complex query or some sort of parametrisation in executed queries. In input 3 the per row iteration model for CSV les implemented in ShExML does not create collections e ectively. Therefore, it would imply a reimplementation of per row iteration model for these cases. However, it could a ect the overall functionality for CSV les. 6
In Q3 we have posed a question about the possible e ects that the modi cations in ShExML could have in already working features. The idea of this question is to demonstrate the validity of Q1 solutions alongside old features that should still work as expected. This type of testing, known as regression testing, have been included in ShExML from the very beginning20 so we are able to add new features in ShExML knowing that old features are still working as expected. Thus, every time a new version is released these tests must be executed to validate the language and engine integrity. Continuous integration is the perfect tool for this task, as every time that a change is submitted to ShExML repository all tests are executed to verify the integrity. In the ShExML repository we have con gured Travis CI21 for this task. Therefore, these regression tests in v0.2.422 and v0.2.523 are telling us that all features are still working as expected, and equally, giving a negative answer to Q3. So, we can conclude that integrity is held.
In Sections 3 and 4 we have seen how some mapping challenges were already
solved in ShExML and how we have made some modi cations in ShExML
language and engine to deal with the others. These two sections give an answer
20 To see all tests that are executed over ShExML engine
https://github.com/herminiogg/ShExML/tree/master/src/test/scala-2.12/es/
weso/shexml
21 https://travis-ci.org/
22 https://travis-ci.org/github/herminiogg/ShExML/builds/755033209
23 https://travis-ci.org/github/herminiogg/ShExML/builds/756419674
for Q1. These solutions were designed to maintain ShExML usability [
In Section 5 we have given some intuition on how remaining challenges could be solved, answering to Q2. They would require harder and more complex modications; in some cases the modi cation of an already working mechanism (e.g., inputs 2 and 3 in RDF Collections), the inclusion of a new iteration model and the design of a new query language (e.g., Excel style) or the choice between two di erent systems (e.g., data transformation functions in process multivalue references). All these inclusions will require a careful study and implementation in the language so they do not a ect other features and, also, to select the better option from a usability perspective. 7
In this paper we have explored how ShExML can deal with some of the challenges de ned in the Knowledge Graph Construction W3C Community Group. We have divided them into challenges already solved by ShExML before their de nition, challenges solved by latest versions of ShExML, and challenges that are not yet solved, for which we have given some notions and intuitions on how ShExML can be modi ed to cover them. Furthermore, we have demonstrated that the modi cation of ShExML to cover new challenges has not a ected other language and engine features. Therefore, we see this work as a rst step on demonstrating how the challenges can be solved and, together with solutions from other languages and the joint discussion, we will be able to o er uni ed solutions to the posed mapping challenges.