This paper describes an extension to the M-Gov framework that captures queryable metadata about matcher tools that have been utilized, the users involved, and the discussions of the users, during the generation of alignments. This increases the traceability in an alignment creation process and enables an evaluator to more deeply interpret and evaluate an alignment, e.g. for reuse or maintenance. This requires precise information about the alignments being encoded and the decisions undertaken during their creation. This information is not captured by state of the art approaches in a queryable format. The paper also describes an experiment that was undertaken to examine the effectiveness of our approach in enabling the traceability in the alignment creation process. In the experiment, stakeholders created an alignment between two different datasets. The results indicate that the users were 93% accurate while creating the alignment. The major traceability achievements demonstrated for the test groups were 1) level of participation of various users of a group during alignment creation; 2) most discussed correspondences by users of a group; and 3) accuracy of a group in creating alignment.
Ontology mapping is required to overcome the problem of semantic heterogeneity and
facilitate interoperability between ontology-based systems that share the same
concepts but have the different representation of those concepts [
Alignments are built for a purpose like data integration or a link data mashup for a
specific group of stakeholders. Creation of an alignment is a non-trivial task, as it
requires these stakeholders to collaborate. In [
Traceability in [
In this paper, we first describe how we have extended the M-Gov framework by supporting stakeholders during the Match phase (Section 3). First, the Alignment API 4.8 is used to discover candidate correspondences between two different datasets. Then stakeholders are allowed to discuss each identified correspondence displayed on a web page using a grid table. The paper also describes (Section 4) the initial evaluation that we have undertaken. Specifically, the research question under investigation during our evaluation was to what extent captured metadata allows tracing of: the most discussed correspondences by stakeholders, the level of participation of stakeholders, and the decisions taken by a group of stakeholders for a correspondence?
In summary, the contribution of this paper is as follows: Firstly by extending the M-Gov framework to enable tracebility in an alingment creation process. Secondly, we have provided a detailed description of the alignment creation process. Thirdly we have provided evidence that metadata captured in the M-Gov framework enables traceability in an alignment creation process.
The paper is organized as follows: Section 2 provides an overview of the background information; Section 3 outlines the match phase of the M-Gov framework; Section 4 presents an evaluation of the experiment that was undertaken; Section 5 sheds some light on the related work; and conclusions are drawn in section 6.
This section presents necessary background on collaborative ontology engineering, community-driven ontology matching and an overview of the M-Gov framework.
Ontology engineering refers to the study of the activities related to the ontology
development, the ontology life cycle, and tools and technologies for building the
ontologies [
In collaborative ontology-engineering, publishing the new version of an ontology
is different to a centralized situation, as there is a need to synchronize the editing. To
facilitate the editing, web-based or desktop based applications are used, and versions
of ontologies are traced with the help of distributed versioning software [
In contrast, our approach does not use distributed versioning software for traceability during alignment creation. M-Gov itself keeps track of each activity that occurs in an alignment creation process.
Community-driven ontology matching (CDOM) extends conventional ontology
matching by involving the community (end users, knowledge engineers, and
developers) in the creation, description, and reuse of mappings [
A prototype has been implemented and analyzed in [
In contrast, M-Gov captures each activity that occurs during alignment creation. The captured information could serve as the rationale for the creation of a mapping. It also allows one to facilitate the discovery and reuse of existing alignments with the help of queries and thus making the alignment creation process more traceable.
Governance refers to [
[
A project-centric perspective has been adopted by [
Adopting a project-centric perspective in ontology mapping process allows one to capture the metadata of various aspects of the mapping process. Using the extension of PROV-O as metadata model makes the ontology mapping process more traceable, as it will not only allow one to formulate queries to reuse existing mappings but also formulate questions about the activities happened during the mapping process.
This paper is built on [
This paper extends the initial M-Gov implementation; it implements the “match phase” of M-Gov and evaluates the correspondences identified in match phase. The next section presents the methodology adopted for ontology matching and evaluation of correspondences.
This section describes the requirements, design, and implementation of the match phase newly developed for the M-Gov framework.
The main objective of the Match Phase is to identify the potential correspondences
between two datasets automatically and capture the metadata produced during the
alignment creation [
To fulfill the functional requirements, there needed to be a number of aspects designed. In this section, we present a quick overview of the design. The design was focused on an initial baseline without sophisticated UI as our focus was on interaction process and capturing of discussions. Future work will develop the UI. In addition, we focused on an alignment problem where pre-processing is not necessary, as the experimental focus was on traceability of the captured discussions. However, it would be easy to add further steps and linked discussions in the M-Gov framework.
A web based form was built to allow the users to configure the matcher by
selecting a source and target ontology, and a matcher tool. The matcher configuration was
stored in a database. Selected ontologies were matched using Alignment API 4.8. A
REST call was designed for communicating with the Alignment API. The Alignment
API returns the potential correspondences in alignment format (an XML format as
shown in Fig. 2.), which was used to capture the M-Gov metadata about the identified
correspondences. The captured metadata is again stored in the database. Furthermore,
an interface was designed to present the M-Gov metadata about the potential
correspondences for stakeholders to discuss. To provide context for discussions about the
correspondences, the values of object1 and object2 on the interface were linked to
their online Linked Data resources. The interface was also designed to show the
comments of all the stakeholders on a correspondence. Thus, allows the stakeholders
to see other perspectives about the fitness of a correspondence. The discussions of
stakeholders are structured by using the IBIS framework and the metadata model used
in the M-Gov framework is an extension of PROV-O, as suggested by [
The capture of discussions was the major challenge faced while designing the MGov match phase supports, as this will enable the traceability in an alignment creation process. For this, we capture every statement given by each stakeholder during the alignment creation. In M-Gov every statement is linked with its creator, and correspondence’s ID on which the statement has been made. M-Gov also captures the conclusion and the stakeholder’s ID who concluded that discussion. Table 2 describes the M-Gov metadata used to track the discussion.
Description Unique identifier attached to each discussion Type of discussion: a conclusion or just an opinion Stakeholder who made the statement Content of statement Type of statement, e.g.: supporting or objecting Final statement while concluding the correspondence Timestamp of the conclusion Stakeholder who concluded the correspondence
If the correspondence is accepted to rejected A form has been built to allow a user to select a source and target ontology, and a matcher tool. A user can select these parameters from a drop-down menu to configure the matcher. The M-Gov uses these parameters to create the URL to invoke a REST call to Alignment API. Fig. 2 describes the response from Alignment API, it shows an example of a potential equivalence correspondence (line 5) between “HumanActor” (line 3) and “HumanActorAge” (line 4) with a confidence of 0.93 (line 6).
The M-Gov displays every potential correspondence on a webpage using grid tables, which also contains a "state" column, whose default value is “inDiscussion”. The M-Gov also attaches a “change decision” button to every displayed correspondence, which is used to start a new discussion thread for that correspondence. If the discussion thread is already created then this button will lead to the in progress discussion for that correspondence. Once the users reach a consensus after discussion, the M-Gov provides a “Conclude discussion” link, which allows a user to change the state of the correspondence to either “Accepted” or “Rejected”. The M-Gov also stores the discussions along with the user’s information under the “post” table in the database.
Fig. 3 represents the page by which stakeholders can add their arguments to participate in a discussion about a correspondence. In our example of Fig. 2, this would involve discussion of whether HumanActor and HumanActorAge are really equivalent? Fig. 3 shows the overview of the correspondence and arguments about its fitness. “reply” textbox can be used to add arguments, while a suitable reply type needs to be selected from the dropdown “Reply Type”, whose values are “Supporting example, objecting example, supporting justification, objecting justification, supporting motivation and objecting motivation”.
Motivation. The purpose of this experiment was to trace the discussions among the stakeholders during the alignment creation process and identify the following: 1) level of participation of various users of a group during alignment creation. 2) most discussed correspondences by users of a group. 3) accuracy of a group in creating alignment.
In the experiment, we have used three types of correspondences: a) Correct correspondences - those in which both objects point towards the same resource. b) Incorrect correspondences - those in which both objects point towards completely different resources. c) Ambiguous correspondences - those in which both objects point towards different resources. But to understand the difference, a user needs to go through a substantial amount of information, as the difference might not be clear from the label of the entities.
Hypothesis. In most cases, the discussion thread attached to an ambiguous correspondence will be longer than correct and incorrect correspondences.
Experiment method. We formed 4 groups, 3 groups contained 3 users while 1 group contained only 2 users. A separate instance of the framework was provided for each group. Every user was located at a different workstation and was allocated discrete credentials to log into the framework. We have only used instance level correspondences in the experiment, since creating concept level correspondences requires participants with a deeper understanding (who are harder to recruit). It was thus decided to first investigate stakeholder collaboration tracing using instance level correspondences, which could be performed by a wider range of participants. We created a discrete set of 7 instance level equivalence correspondences for each group, the complete list is available online1. Semantic mapping researchers validated the created correspondences. These correspondences have been created manually and injected in the framework for discussion, which covers three types of correspondences as follows: a) Correct correspondence: These are created by taking an entity from OSi2 dataset as object1, while the object2 has been selected from DBpedia3, which points to the exact same resource as referred by object1. For example, “County Roscommon represented by OSi” and “County Roscommon represented by DBpedia”, b) Incorrect correspondence: These are created by taking an entity from OSi dataset as object1, while the object2 has been selected from DBpedia, which points to a completely different resource than that referred by object1. For example, “County Roscommon represented by OSi” and “County Clare represented by DBpedia”, c) Ambiguous correspondence: These are created by taking an entity from OSi dataset as object1, while the object2 has been selected from DBpedia, which points to the resource that has a label similar to the resource referred by object1. To figure out the difference between both objects, a user needs to examine the available information about both the resources. For example, “County Tipperary represented by OSi” and “Tipperary town represented by DBpedia”. Participants can discuss the correspondences within the group only through the framework. For deciding upon a correspondence, if it was acceptable or not, users needed to come to a consensus.
Metrics. To trace the most discussed correspondences in a group, the word count
in the statements of the users will be used to calculate the length of the discussion.
The word count for a discussion in a group also depends on the active users in a
group. At the end of the experiment, users will be asked to evaluate the use of the
framework by providing answers to usability based questions of PSSUQ [
Datasets. A subset of entities in the OSi county dataset and in the DBpedia dataset for counties of the Republic of Ireland has been used to create correspondences.
User recruitment. The selected users were M.Sc. students of computer science at Trinity College Dublin. For preparing the users for the experiment, we have given a presentation, a video tutorial and a detailed version of user instructions to users about how to use the M-Gov to curate the correspondences. All the documents related to the experiment preparation are available online4.
Data analysis. For each group, Fig. 4 describes the type of correspondence and length of discussion involved in coming to the conclusion. Fig. 5 describes the individual contribution of the users in each group. Group 1 had 3 users: user 9, 10, and 11. However, user 10 did not participate in the discussion properly. For group 1, the 1 https://github.com/anujsinghdm/Experiment/blob/master/AllCorrepondence.xlsx 2 http://data.geohive.ie/ 3 http://wiki.dbpedia.org/ 4 https://github.com/anujsinghdm/Experiment/tree/master/UserInstruction longest discussion thread has been attached to C4, which is a correct correspondence and it is also clear from Fig. 4 and 5, user 9 and 11 were mostly discussing the nonambiguous correspondences, hence group 1 does not support the hypothesis.
Group 2 had 2 users: user 7 and 8. However, the majority of the word count represents the user 8. For group 2, the longest discussion threads have been attached to C1 and C4, where C1 is the correct correspondence, while the C4 is the ambiguous correspondence. Users did not discuss much the 2nd ambiguous correspondence, only user 8 gave the statement, why it wants to reject the correspondence. Having the discussions analyzed, we can say that user 7 did not participate in the discussions properly and group 2 is also not in support of the hypothesis.
Group 3 had 3 users: user 4, 5, and 6. For group 3, the longest discussion thread has been attached to an ambiguous correspondence C4. Fig. 4 describes that group 3 discussed incorrect correspondences more. This might be the reason why the 2nd ambiguous correspondence does not have a longer discussion, as the users perceived it a correct correspondence. Group 3 concluded one more correspondence incorrectly, but we believe that is just an operation error, as the attached discussion indicates that they analyzed the correspondence correctly. Group 3 supports the hypothesis as the longest discussion thread is attached to an ambiguous correspondence.
Group 4 had 3 users: user 1, 2, and 3. However, most of the discussions have been carried out by user 1. As it is clear from Fig. 4, ambiguous correspondences (C4 and C7) have the longest discussion thread attached to them. Hence, group 4 supports the hypothesis.
Fig. 5. Individual contribution of each user in discussing the correspondences
Finally, participants were asked to complete a PSSUQ [
Conclusions. The results indicate that except group 3, every other group was accurate (compared to the gold standard) in creating alignment. Group 3 incorrectly concluded 2 correspondences out of 7. The results also show that group 1 and 2 do not support the hypothesis, however not every member of this group actively took part in the discussion. Group 3 supports the hypothesis as for the 1st ambiguous correspondence, the discussion thread is the longest. We believe that the users of this group miscomprehended the information of 2nd ambiguous correspondence, hence the correspondence did not get discussed in detail and concluded incorrectly. Group 4 clearly supports the hypothesis as they discussed ambiguous correspondences the most. Gathered data is unable to lead us to any conclusion about the hypothesis, as two groups are supporting the hypothesis while the other two groups are not in support of it. However, the results provide an evidence that the captured metadata by M-Gov enabled the traceability in the alignment creation process. Additionaly, for the technical contribution we tracked the following: a) level of participation of users, b) most discussed correspondences and, c) the accuracy of groups in alignment creation. We can also conclude from Fig. 6, user 9 and 4 are the outliers as most of their responses do not comply with other users. The data from the PSSUQ suggests that 72% users were satisfied by using M-Gov but enhancements in terms of UI/UX are required so that tasks could be performed more efficiently.
A variety of approaches has been used to evaluate the methodologies/ frameworks that support collaborative ontology engineering. We see two evaluation approaches related to our work. This section focuses on these approaches.
Domain experts are supported by [
The Ontology development framework proposed in [
The paper presents an extension of M-Gov framework to match the two different datasets automatically and capture the metadata produced during the alignment creation. The paper also describes an experiment in which 11 stakeholders discussed the potential correspondences to create an alignment. The aim was to trace the metadata produced during the alignment creation.
The research question presented in this paper is to what extent the captured metadata allows us to trace the most discussed correspondences by users, the level of participation of users, and the decisions undertaken by a group of users for a correspondence to determine if it is acceptable or not, in an alignment creation process? We also present the evaluation of M-Gov by users in creating alignment.
An experiment was conducted to create an alignment between the locations in DBpedia and OSi dataset. Based on the results, we are unable to conclude the hypothesis, as two groups are supporting it while the other two groups are not in support of the hypothesis. However, it provides an evidence that the captured metadata during the alignment creation enables traceability. In addition to this, the technical contribution of our work involves tracing the following: a) Group 1 and 2 discussed mostly the non-ambiguous correspondences, as the discussion thread attached to nonambiguous correspondences are the longest. Group 3 and 4 have the longest thread attached to the ambiguous correspondences, so group 3 and 4 discussed mostly the ambiguous correspondences. b) Not every participant in group 1 and 2 was actively engaged in the discussion. c) 26 correspondences out of 28 were concluded correctly. We would be able to do more detailed analysis if the participants would have been more active in each group as we would have got richer experiment data. Acknowledgements. This research has received funding from the ADAPT Centre for Digital Content Technology, funded under the SFI Research Centres Programme (Grant 13/RC/2106) and co-funded by the European Regional Development Fund.