Usability has been seen as a new requirement in the Semantic Web community. Heterogeneous data mapping languages enable users to create knowledge graphs from legacy datasets. Even though some of these tools claim to be user friendly, this is not empirically demonstrated. In this paper, we revisit our previous usability experiment with these languages and from its outcomes we envisage next actions and problems that should be tackled in the topic. Covering them should lead to a better adoption among users.
Recent achievements in the data mapping topic enable users to de ne
heterogeneous data sources integration in a declarative fashion instead of using ad-hoc
solutions which redounds in a higher exible and faster process [
Recent trends in the semantic web community have seen the necessity to understand users and put them in the center of our solutions, improving their productivity and taking care of their needs1. Moreover, this has also been highlighted in the Knowledge Graph Construction W3C Community Group2. Users or usability studies allow to understand users' problems as well as their di culties, needs and perceptions. They are, therefore, a huge analysis tool when deciding future actions on the topic.
However, to the best of our knowledge, only our recent study [
Currently, there are various languages that allow to integrate heterogeneous
data sources into a single Knowledge Graph. First comers based their syntax
in the W3C Recommendation R2RML3, like RML [
While having an RDF-based syntax could seem a good thing for machine readability, inference and processability, it is also true that it makes the nal language verbose (even when using the Turtle syntax). In this regard, we could consider these languages more like middle representation languages rather than languages intended to be used directly by users. Following this idea, including them into a user study would not be fair, as users would need more time and keystrokes to achieve the same solution. Therefore, it would be a bias from the beginning, as these users would not be in the same conditions, in comparison with other users assigned with much less verbose languages. Thus, we selected those languages that claim to be user friendly and that o er a similar syntax in terms of verbosity.
The experiment was designed as a mixed-method approach, that is to say,
involving a quantitative and a qualitative design and analysis [
The sample consisted of 20 students pursuing a MSc in Web Engineering ( rst course out of two). The experiment was hosted the nal day of the semantic web subject in which the students were introduced to semantic technologies (RDF, SPARQL, Shape Expressions, among others). Therefore, we can categorise the sample as rst-time users with some background knowledge. So, our results and conclusions will be in line with this described pro le. 3 https://www.w3.org/TR/r2rml/
The sample was randomly distributed in three groups, one per language, so previous knowledge background bias could be mitigated. The experiment was divided in two tasks. The rst one consisted in creating a set of mapping rules from a given input and the expected output. The second one was to perform a small modi cation to the mapping rules created in the former task. Therefore, rst task measured global usability whereas the second one measured modi ability of the mapping rules developed with the assigned language. 3
In the rst task, 17 students (out of 20) submitted results (7 for ShExML, 4 for
YARRRML, and 4 for SPARQL-Generate) and in the second task only 7 students
did so (6 for ShExML and 1 for YARRRML). These total results reveal that the
SPARQL-Generate users had problems when reaching a working mapping and
that the YARRRML users found di cult to modify an existing set of mapping
rules and/or that they invested too much time in the rst task. We performed a
statistical analysis (cf. [
Task 1: In quantitative analysis signi cant di erences were found in elapsed seconds (particularly between ShExML and YARRRML), completeness percentage (between ShExML and SPARQL-Generate) and precision (between ShExML and SPARQL-Generate). This comes to corroborate that the YARRRML users invested much more time than the ShExML users when nding working solutions. The di erence in completeness percentage and precision between ShExML and SPARQL-Generate reveals that the SPARQL-Generate users were not able to nd working solutions.
It is also worth to mention that there were no signi cant di erences in the rest of quantitative variables which come to support that there are no di erences in languages verbosity (i.e., no signi cant di erences in keystrokes) and in the o ered web playground, which was used by the users to develop the mapping rules (i.e., left button clicks, right button clicks, mouse wheel scroll and meters travelled by the mouse). This is of quite importance as, like we mentioned in Section 2, a di erence in languages verbosity would imply much more time invested by users to reach a working solution. In the same way, a signi cant difference in user interfaces could lead to the similar biases when solely analysing the languages usability.
In qualitative analysis signi cant di erences were found in general satisfaction (between ShExML and YARRRML), learnability (between ShExML and other both languages), mapping de nition easiness (between ShExML and both other languages) and easiness of use (between ShExML and YARRRML). These qualitative results come to corroborate and complement the quantitative ones, so di culties in nding working solutions by the SPARQL-Generate users are translated to a worse learnability and mapping de nitions easiness impression. More time consumed to nd working solutions by the YARRRML users is translated to a worse impression in the four variables causing a descent in general usability indicators (i.e., general satisfaction level and easiness of use).
Task 2: In task 2 no signi cant di erences could be established due to the very low sample sizes. Only one YARRRML user was able to submit a nonworking solution whereas 6 ShExML users submitted a solution for this task. This could be caused by the extra time needed by the YARRRML users wrt the ShExML users. The ShExML modi ability variable was rated with 5 points by 83% of the ShExML users and with 3 points by the YARRRML single user. The SPARQL-Generate users were unable to reach this task as they had problems nishing the rst one.
These di erences reveal that the design of SPARQL-Generate is having a bad e ect on rst-time users which found it hard to operate and learn. However, it would be interesting to discern which parts of the languages are causing the di erences between ShExML and YARRRML. As an hypothesis, we can explain them due to their di erent syntaxes because ShExML uses keywords which can make the language more self-explanatory and o ers modularity in its iterators, which reminds the, well-known by developers, object-oriented paradigm.
Bad results in some qualitative variables for the three languages reveal another interesting picture. They perceive that the languages design lead to commit some errors (error proneness), that the error reporting system was not useful to solve their errors (error reporting system) and that they do not see much applicability to these tools (applicability). As we analyse further in the following section, these three aspects should be handled urgently by the community. 4
In the light of the previously commented results it is important how new features are added and designed so they do not have a bad impact on usability and learnability. In the semantic web community we care a lot about new features and technical improvements but we tend to care less about users, we should involve them more and develop more user-centric approaches.
Following the previous argument, the users from our experiment told us that the three languages lead them to commit some errors (the languages are not designed taking users mental models into account), that the error reporting system was not useful (it is another point that reveals that users are not involved in the development process) and that they do not see much applicability in these tools. This last perception reveals something that could be extended to the whole semantic web community. With these languages and tools we are producing knowledge graphs, so in the end they do not see applicability to neither of them. In addition, applicability (and related variables like learnability) on rst-time users reveal a derivate and correlated one: adoption. If rst-time users do not see much applicability, and technologies are hard to learn, they are not going to adopt them. Therefore, these points are urgent ones that should be addressed by the community.
From a methodological point of view, we have to adopt stronger methods [
As we mentioned, our study only covered rst-time users with some
background knowledge so it is necessary to run these kinds of experiments with other
pro les to have a whole perspective on the topic [
Finally, in order to contrast our hypothesis about di erences between ShExML
and YARRRML it would involve running more complex experiments which could
come closer to the users' mental model processes, so we can understand which
language constructions and syntax are better. One possibility is to use
cognitive models and frameworks [
The Semantic Web community recent trend to focus on users, understand them, and take care of their needs has opened another perspective in the semantic technological stack which advocates to put the user in the center of our thoughts. Although some heterogeneous data mapping languages claimed its user friendliness, it was not empirically supported. Thus, in this paper we have brie y summarised our previous usability experiment in the eld of heterogeneous data mapping languages where ShExML demonstrated a better usability on rst-time users.
We have also claimed actions that should be addressed to solve the elucidated common problems, further experiments that cover more users' pro les, 4 https://slideshare.net/miriamfs/vision-track-october2020fernandezv5 5 https://slideshare.net/tammavalentina/the-tao-of-knowledge-the-journey-vs-thegoal more complex experiments that could explain users' mental model processes and the use of stronger methodological instruments and metrics. Dealing with the exposed points, we envisage a promising future for the Knowledge Graph Construction community, decreasing their technologies complexity barriers, having more users being attracted, and in short, improving their adoption.