This extended abstract summarizes a rigorous investigation about the effectiveness of the impact of wiki collaborative functionalities on the collaborative ontology authoring. The work summarized in this extended abstract has been published in [DGR14]. Context. This extended abstract summarizes a rigorous investigation about the impact of wiki collaborative functionalities on ontology modelling, presented in: Evaluating Wiki Collaborative Features in Ontology Authoring Di Francescomarino, C.; Ghidini, C.; Rospocher, M.; IEEE Transaction on Knowledge and Data Engineering, 26 (12): 2997-3011, 2014. Good quality ontology modelling often demands for multiple competencies and skills, which are difficult to find in a single person. This results in the need of involving more actors, possibly with different roles and expertise, collaborating towards the ontology construction. Collaborative ontology authoring has been recently widely investigated in the literature [Su02, Tu10, Di08, Ro14, RTM14]. These studies show that the collaboration among the different involved actors needs to be fostered by taking into account two important requirements. A first requirement deals with the collaboration between who knows the domain that is going to be modelled, i.e., the Domain Expert (DE) and who has the technical skills to formalize the domain modelling. i.e., the Knowledge Engineer (KE). Traditional methodologies and tools were mainly based on the idea that knowledge engineers should drive the modelling process (producing ontologies in a formalism which is usually not understandable for domain experts) and domain experts should only report to KEs their knowledge of the domain. However, these methodologies often create an unnecessary extra layer of indirectness, an imbalance between the two roles and the impossibility for the domain experts to understand the modelled ontology. DEs should be actively involved in the ontology modelling process rather than only provide domain knowledge to KEs. A second important requirement deals with the support of distributed teams of actors. Independently of their geographical position or their role, team members should be made aware about the collaborative development of the modelled artefacts, should be supported in the communication of modeling choices, as well as in the work coordination. Wiki tools for the ontology authoring offer an appealing option for tackling these collaborative aspects. Indeed wikis usually provide collaborative features (wiki collaborative
features) useful for supporting ontology authoring, such as (i) easily customizable interfaces for the ontology editing; and (ii) functionalities for discussing ideas, commenting decisions, notifying and tracking changes.
This extended abstract summarizes the work described in detail in [DGR14, DGR12]. Purpose of the work is offering a rigorous investigation, based on a theoretical and empirical evaluation, of the effectiveness of the impact of wiki functionalities on the collaborative ontology authoring. The results show that wiki collaborative features, by actively involving domain experts in the authoring process and reducing the interaction effort of modellers, support and affect the process of collaborative ontology authoring, as well as the lifecycle of the built ontology entities.
Wiki Collaborative Features. Wiki systems, originally created for unstructured content like the Wikipedia pages, have been evolved towards wiki tools supporting the collaborative authoring of structured content, including ontologies [ADR06, GRS12]. In this work we focus on the following collaborative functionalities.
Multi-mode access to content: interfaces for accessing or editing content can be easily customized according to roles and users; Discussion mechanism: wikis provide mechanisms to enable users to communicate, offering support for the creation of discussion threads, sections and replies;
Theoretical Evaluation. It is based on a framework for the analysis of collaboration features in process modelling tools [MRW12]. The framework evaluates the technical support provided by tools with respect to the five levels of social interaction [MC94]: awareness, communication, coordination, group decision making and team-building. By applying the framework to the wiki collaborative features of a wiki tool [GRS12] and investigating their support to each of the interaction aspects, we found that each of the features contributes to support one or more of the social interaction aspects and that all the five levels are actually covered by these features:
Awareness Revision history, Watchlist, Notification mechanism, Multi-mode access to content, (Shared workspace);
Empirical Evaluation. For a rigorous evaluation of the impact of the wiki collaborative features on the collaborative ontology authoring, we carefully designed the empirical evaluation. Specifically, we focused on the evaluation of (i) the effectiveness of the wiki collaborative features in the process of ontology modelling; (ii) the impact of collaborative features on the collaborative ontology modelling and on the (ontology) entity lifecyle. For each of these aspects we further refined the analysis by defining specific research questions. In detail, for the first aspect we looked at the effectiveness of the features based on three different factors, i.e., DEs involvement (RQ1), required effort (RQ2) and designers’ perception (RQ3). Concerning the impact of the collaborative features, we separately focused on the ontology modelling process (RQ4) and on the entity lifecycle process (RQ5). The empirical study carried out for answering the research questions involved four teams, each composed of a DE and two KEs. Each team was asked to collaboratively model ontologies for two domains from the pedagogical field. To this end, all the teams members were provided with a modelling tool [GRS12], as well as with chat and email support. Specifically, each team was asked to model the ontology of one of the domains with the modelling tool enhanced with the wiki collaborative features (CMT- Collaborative Modelling Tool), and the ontology related to the other domain with the same modelling tool in which the wiki collaborative features were removed (NCMT- Non-Collaborative Modelling Tool). The four teams carried out the modelling task on the two domains in two laboratory sessions, one with CMT and the other with NCMT according to a balanced design [Wo00]. Moreover, each session was divided in 5 different phases: the first four phases aimed at simulating an asynchronous collaboration, i.e., DEs and the KE working in different (alternate) phases; the last phase was instead focused on the synchronous collaboration, i.e., all the three team members working together.
From the analysis of the results we found that the wiki collaborative features actually increase the involvement of DEs in the ontology editing (RQ1). Indeed, when CMT is used, the DEs average number of edited axioms and of editing operations is statistically significantly greater than when NCMT is used. An inverse pattern occurs instead for KEs, i.e., the KEs average number of edited axioms and of editing operations with CMT is statistically significantly lower than with CMT. Concerning RQ2, the results confirm that the wiki collaborative features reduce the effort required by team members to communicate. Indeed, the length of the (chat and email) conversations carried out with CMT is statistically significantly lower than with NCMT. Overall modellers showed a (statistically significant) positive evaluation about the perceived overall effectiveness and ease of use of the wiki collaborative functionalities, while they were a bit more skeptical with respect to the effectiveness of the features in addressing some of the five interaction levels, i.e., support for coordination and team building (RQ3). The obtained results also show that the collaborative functionalities have an impact on the ontology modelling process (RQ4). Specifically, it comes out that the process of ontology authoring carried out with CMT is less rigid and encourages more agile interactions among team members than the one carried out with NCMT. By further inspecting this result, we were able to (statistically) confirm that a high usage of the wiki collaborative functionalities correspond to a high dynamism in the typology of activities carried out. Finally, also the created ontology entity lifecycle seems to be affected by the usage of the collaborative functionalities (RQ5). Qualitative differences between the lifecycle of CMT and NCMT ontology entities seems to reveal that CMT entities are built according to a process more structured than the one used for NCMT entities, thus suggesting a possibly higher quality of the resulting CMT entities.