R2RML is a W3C Recommendation that provides for the declaration of mappings to generate RDF datasets from relational databases. One issue that hampers its adoption is the manual effort needed in the creation and maintenance of such mappings. To tackle this problem, various initiatives have started to emerge. One of the directions is to investigate how different representations can facilitate the creation and maintenance of such mappings for a wider set of stakeholders. In prior work, we proposed a visual representation based on the block metaphor for R2RML mappings that is compliant with this specification. This representation has been integrated within a tool for creating and managing R2RML mappings. In this paper, we report on a user study to evaluate the proposed visual representation considering stakeholders with different background knowledge. Preliminary findings indicate that participants were able to create accurate mappings and that the visual representation achieves good results in standard usability evaluations.
A huge part of the Linked Data web is achieved by converting non-RDF resources
into RDF. This conversion process is typically called uplift. For relational databases,
one can rely on the W3C Recommendation R2RML [
Initiatives have emerged to address these problems and make the technology more
accessible ranging from step-by-step wizards [
In previous work, we have proposed a visual representation for mappings, Juma
[
In this paper, we present a user study that evaluates our visual representation of mappings applied to the R2RML mapping language considering different types of stakeholders. Our intuition is that the visual representation would be useful for nonexperts and experts in Linked Data publishing from relational databases. This user study evaluated the mappings created by participants and the usability of the visual representation through a standard usability test.
The remainder of this paper is structured as follows: Section 2 reviews the related work. In Section 3 we discuss the R2RML mapping language. Section 4 describes Juma. Section 5 presents a user study used to evaluate our visual representation. Results and analysis are presented in Section 6. Section 7 concludes the paper. 2
In this section, we discuss the state-of-the-art in mapping representation for R2RML. We have characterized these into applications with or without visual representations. Applications without visual representations have an interface that guides users in the creation of R2RML mappings. These tools, however, do not provide any visual representation for mappings. Applications with a visual representation offer a graphical view of the mapping.
No visual representation. The fluidOps editor [
Visual representation. Karma [
Visual representations are especially helpful to non-expert users. However these
usually focus on Knowledge Engineers, representing uplift mappings as graphs, since
the RDF data model is itself one. This representation, nonetheless, is not as intuitive
for other types of user. As mentioned before, the block metaphor has been
successfully used in other domains to attract a range of stakeholders (e.g. programming).
Furthermore, in [
R2RML
In this section, we briefly explain the main concepts related to the W3C
Recommendation R2RML for the purpose of this paper. For more information, we refer the
reader to the W3C Recommendation [
Listing 1. R2RML mapping definition
In this mapping, we map the table (or view) “students”. A triples map defines subjects to have the IRI http://example.org/student/{id}. We also declare subjects to be instances of the class foaf:Person. A predicate object map relates the subjects with the predicate foaf:name to values in the column “name” . 4
In previous work we have presented a method called Jigsaw puzzles for representing
mappings, Juma [
Our implementation uses Google’s Blockly API5. Google Blockly is a visual programming language that uses blocks to facilitate code creation. These blocks are shaped like jigsaw puzzle pieces that show how the language works by abstracting the language’s syntax. Furthermore, Blockly has been successfully used in many projects, such as code.org’s6 introduction courses to Computer Science.
In our tool, each block has been designed to represent an R2RML statement that automatically generates a correspondent R2RML construct. The visual representation abstracts the R2RML vocabulary’s syntax for users. The visual representation also guides users in the creation of valid mappings by highlighting and only allowing the connection of blocks that would create a valid mapping. The menu options provide one with all possibilities within the R2RML mapping language, from which new blocks may be dragged into the workspace. The menu is also defined using a tree structure, so as to gives users a hint of how the blocks connect to each other. The visual representation also uses colors to identity the type of structure that is being created. For example, all constructs for subjects are in green, predicates are blue and so on. Furthermore, the template menu option shows a complete triple map and a predicate object map, with the default R2RML options, that can be used to bootstrap mapping creation.
For each mapping there are 3 tabs. In the first tab, Mapping, we show the menu and the visual representation. In Configuration, one can define the properties of the configuration file. The configuration file is used as input to an R2RML processor together with the R2RML mapping file. In the R2RML-Mapping tab the user can see the actual R2RML mapping generated from the visual representation. Fig. 1 shows the R2RML mapping from Listing 1 represented in Juma. 4 https://www.scss.tcd.ie/~crottija/juma/ accessed in August 2017. 5 https://developers.google.com/blockly/ accessed in August 2017. 6 https://code.org/about accessed in August 2017.
Participants were recruited based on their background knowledge. These groups were chosen as to evaluate how different types of stakeholders engage with our visual representation in the creation of uplift mappings. We decided to focus on users that are likely to need to publish Linked Data datasets. Our intuition is that the visual representation would be useful for non-experts and experts in Linked Data publishing from relational databases. We consider that these users would be one of three types: non Semantic Web experts with background in computer science but no knowledge of R2RML; Semantic Web experts with no knowledge of R2RML; and Semantic Web experts with experience of the R2RML mapping language for Linked Data publishing. The groups were defined as follows: • Web Developers (WD): these participants had background in computer science with experience on web development, but not in Semantic Web technologies or on the R2RML mapping language; • Knowledge Engineers (KE): these participants had knowledge in Semantic Web technologies such as RDF, OWL and so on. Furthermore, these users would also not be familiar with R2RML; • R2RML familiar (RF): participants in this group had experience with R2RML. 5.2
The study was structured in four parts: 1. Pre-task questionnaire: participants were asked to evaluate their knowledge in relevant fields (Semantic Web technologies and more specifically about the R2RML mapping language). Participants evaluated their familiarity using a 7-point Likert scale from 1 (strongly agree) to 7 (strongly disagree). If the participant’s response was in the range 1 to 4, they were considered familiar with the technology. 2. Technical debriefing: after filling out the pre-questionnaire, participants had the opportunity to watch videos about RDF, R2RML and our tool. If they felt comfortable with these technologies, they could skip the videos. A presentation explaining how the tool works was also available to be used during the experiment7. They could watch the videos and use the presentation during the task. 3. Mapping task: in the main part of this study, we asked participants to create one R2RML mapping using the tool. The task is described in next section. Participants could ask questions to clarify any doubts about the experiment. In addition, they were advised to use the material provided. Any help needed to solve the task was recorded. 4. Post-task questionnaire: after completion of the task, we asked participants to fill out a questionnaire about the use of the tool. At this stage, we have also conducted an informal interview with participants. 5.3
This user study was built on top of the Microsoft Access 2010 Northwind sample database for MySQL8.
Participants were asked to create one R2RML mapping in three parts. For each
part, a sample RDF output was shown to participants. In addition, they could run the
mapping and compare the output from the tool and the sample provided. In this sense,
for the purpose of this experiment, we integrated an R2RML processor [
The task involved the use of different R2RML constructs, such as parent triples maps and others, as to explore their visual representation within the tool. Some elements of the task could be achieved in different ways. For example, since not all attributes are mapped, participants could use a SQL query instead of mapping the whole table, especially for Part 2, which maps only one column. Concatenating could be implemented using a template construct or an SQL query. The template construct would be the expected solution to concatenating. Part 3 of the experiment asked participants to relate the subjects created in Part 1 and Part 2. This could be achieved with a parent triples map or a template construct, since this value comes from the same table. For Part 3, parent triples map would be the expected solution. Table 1 shows the challenges associated to the task. The study was executed with 15 participants, 5 in each group, thus 10 participants have no knowledge of R2RML, as defined in Section 5.1. The experiment was executed individually with each participant in a 13” MacBook Pro (2560 x 1600 resolution). This section will discuss the results and analysis arising from the experiment under two headings: the execution of the task and the usability of the visual representation. In this section we show the data collected during the execution of the task. These include the mappings created, the time taken to execute the task, and any help needed by participants. Table 2 shows accuracy and time by participant. • Accuracy: the mappings created by the participants were executed and compared against an expected output. We calculate accuracy by the number of correct triples in the RDF output. In this sense, a correct mapping would have 53 triples over 9 records. Part 1 has 36 triples. Part 2 has 8 triples and Part 3, 9 triples. The total number of correct triples is indicated in the header of the table. We used Jena API to compare the RDF models and count the number of triples. Syntactic mistakes, such as missing slashes and extra/missing spaces, were not considered errors; • Time: the time taken to execute the task in minutes for each part and for the task.
We recorded the time manually as participants indicated that they have finished each part of the task; • Help: during the experiment, some participants needed help in the execution of the task. In Part 1, 2 Web developers and 1 Knowledge engineer needed help. The only help needed in Part 1 was on concatenating two columns. This can be done using an R2RML template construct or by mapping using an SQL query. In Part 2, none of the participants needed help. In Part 3, 4 Web developers and 2 Knowledge engineers needed help to interlink the subjects created in Part 1 and Part 2. Group
WD KE RF
The mapping accuracy between all participants and within their own groups was high. The R2RML familiar group had the highest score. Moreover, participants from the
R2RML familiar group did not need help to complete the task. This may be explained by the naming conventions used in the tool, following R2RML’s mapping language.
Considering the time taken to execute the task, Web developers spent significantly more time than the other groups. The biggest difference is in the execution of Part 1, which indicates a higher learning curve for participants that are not familiar with Semantic Web technologies.
The most common help was on how to interlink triples maps with the use of the parent triples map construct. Participants were able to create the R2RML construct using the tool but had difficulties defining the parent and child values for the join condition, which requires knowledge on SQL joins. In this sense, the tool offered some support for the creation of these constructs, but some participants struggled with the conceptualization of it. 6.2
Participants were asked to fill in the Post-Study System Usability Questionnaire
(PSSUQ) questionnaire [
The PSSUQ is a 19-item questionnaire with a 7-point Likert scale from 1 (strongly
agree) to 7 (strongly disagree), with a not applicable option (N/A) and a comment
area per question. PSSUQ gives scores in four categories: System Usefulness
(SysUse), Information Quality (InfoQua), Interface Quality (IntQua) and Overall [
Fig. 2 shows a boxplot of all PSSUQs’ responses. We can see that we have one outlier in System Usefulness and two in Information Quality and Overall scores. These outliers are 2 participants from the Knowledge Engineer group. We also used the Welch Two Sample t-test between the groups in every aspect of the PSSUQ questionnaire to check for any significant differences. Table 5 shows the p-values for this test.
All p-values are above 0.05. This suggests that the differences between the groups are not significant. Due to the sample size, we have also applied the Friedman nonparametric test, from which the same conclusion was drawn.
Most participants did not leave any comments. Three of the participants suggested tool improvements such as showing the relational database diagram, and minimizing the typing required by users within the tool (e.g. auto-completion). 6.2.1
We can see that the best average scores are in the R2RML familiar group, as it is shown in Table 4. As mentioned before, this may be explained the naming conventions adopted by the tool.
The scores in the Web developers group are more similar to the R2RML familiar group than to the Knowledge Engineer group. This may be explained by the group’s expectations. In an informal interview made with participants after finishing the task, some mentioned that R2RML is complex and that abstractions in such technologies help with its adoption, especially for non-experts. Knowledge Engineers commented that they expected more information within the visual representation, while participants in the R2RML familiar group were comfortable with the concepts used.
It was also mentioned that the tool works as a template, as one does not need to know all classes and properties of the R2RML vocabulary to create mappings. Moreover, that the system quickly shows the possible constructs, only allowing blocks to connect with others as to create a valid R2RML mapping. In this group, one participant showed concern about the visual representation for large R2RML mappings. The tool offers ways of focusing on smaller parts of the mapping, by collapsing and/or expanding blocks. However, this needs to be evaluated.
In general, the tool received good usability results, within each group and overall, as can be seen in the average scores shown in Table 4. In this table, we can see that interface quality had the best score (1.5); followed by system usefulness and overall (1.9 each); and finally information quality (2.8). In addition, we applied statistical tests to compare the PSSUQ scores between the different groups. As can be seen in Table 5, these differences were not statistically significant. Furthermore, the p-values nearest to the threshold (0.05) involved the outliers identified in Fig. 2. 7
In this paper, we have presented a user experiment to evaluate Juma, a visual representation based on the block metaphor applied to the R2RML mapping language.
We have shown that the visual representation was beneficial in the creation of accurate R2RML mappings for participants with different background knowledge. Furthermore, the most common help needed to complete the task was in the use of parent triples maps. The creation of this construct within the tool was considered difficult for participants with no previous knowledge on R2RML. We have also used a standard usability test to validate the visual representation. The group familiar with R2RML had the highest scores in the usability test, followed by Web developers and Knowledge engineers. As mentioned before in our analysis, this may be explained by the expectations of these users. The usability test also indicated that information quality within the tool was deficient for some users. We believe that improvements in this characteristic will have a positive effect in other usability aspects.
Future work includes incorporating transformation functions, as proposed in [
Acknowledgements. This paper was supported by CNPQ, National Counsel of Technological and Scientific Development – Brazil and by the Science Foundation Ireland (Grant 13/RC/2106) as part of the ADAPT Centre for Digital Content Technology (http://www.adaptcentre.ie/) at Trinity College Dublin.