Jan Mendling and Stefanie Rinderle-Ma, eds.: Proceedings of EMISA 2016, Gesellschaft für Informatik, Bonn 2016 Evaluating Wiki Collaborative Features in Ontology Authoring (Extended abstract) Chiara Di Francescomarino Chiara Ghidini Marco Rospocher1 Abstract: This extended abstract summarizes a rigorous investigation about the effectiveness of the impact of wiki collaborative functionalities on the collaborative ontology authoring. The work summarized in this extended abstract has been published in [DGR14]. Context. This extended abstract summarizes a rigorous investigation about the impact of wiki collaborative functionalities on ontology modelling, presented in: Evaluating Wiki Collaborative Features in Ontology Authoring Di Francescomarino, C.; Ghidini, C.; Rospocher, M.; IEEE Transaction on Knowledge and Data Engineering, 26 (12): 2997-3011, 2014. Good quality ontology modelling often demands for multiple competencies and skills, which are difficult to find in a single person. This results in the need of involving more actors, possibly with different roles and expertise, collaborating towards the ontology con- struction. Collaborative ontology authoring has been recently widely investigated in the literature [Su02, Tu10, Di08, Ro14, RTM14]. These studies show that the collaboration among the different involved actors needs to be fostered by taking into account two im- portant requirements. A first requirement deals with the collaboration between who knows the domain that is going to be modelled, i.e., the Domain Expert (DE) and who has the technical skills to formalize the domain modelling. i.e., the Knowledge Engineer (KE). Traditional method- ologies and tools were mainly based on the idea that knowledge engineers should drive the modelling process (producing ontologies in a formalism which is usually not understand- able for domain experts) and domain experts should only report to KEs their knowledge of the domain. However, these methodologies often create an unnecessary extra layer of indirectness, an imbalance between the two roles and the impossibility for the domain ex- perts to understand the modelled ontology. DEs should be actively involved in the ontology modelling process rather than only provide domain knowledge to KEs. A second important requirement deals with the support of distributed teams of actors. Independently of their geographical position or their role, team members should be made aware about the collaborative development of the modelled artefacts, should be supported in the communication of modeling choices, as well as in the work coordination. Wiki tools for the ontology authoring offer an appealing option for tackling these col- laborative aspects. Indeed wikis usually provide collaborative features (wiki collaborative 1 Fondazione Bruno Kessler, Via Sommarive, 18, 38123 Trento, dfmchiara|ghidini|rospocher@fbk,eu features) useful for supporting ontology authoring, such as (i) easily customizable inter- faces for the ontology editing; and (ii) functionalities for discussing ideas, commenting decisions, notifying and tracking changes. This extended abstract summarizes the work described in detail in [DGR14, DGR12]. Pur- pose of the work is offering a rigorous investigation, based on a theoretical and empirical evaluation, of the effectiveness of the impact of wiki functionalities on the collaborative ontology authoring. The results show that wiki collaborative features, by actively involving domain experts in the authoring process and reducing the interaction effort of modellers, support and affect the process of collaborative ontology authoring, as well as the lifecycle of the built ontology entities. Wiki Collaborative Features. Wiki systems, originally created for unstructured content like the Wikipedia pages, have been evolved towards wiki tools supporting the collabora- tive authoring of structured content, including ontologies [ADR06, GRS12]. In this work we focus on the following collaborative functionalities. • Multi-mode access to content: interfaces for accessing or editing content can be easily customized according to roles and users; • Discussion mechanism: wikis provide mechanisms to enable users to communicate, offering support for the creation of discussion threads, sections and replies; • Watchlist: wikis allow users to monitor the evolution of pages of interest; • Notification mechanism: wikis allow for notification services to the users in case of changes in the pages of interest; • Revision history: wikis offer the complete tracking of any change and comment. Theoretical Evaluation. It is based on a framework for the analysis of collaboration fea- tures in process modelling tools [MRW12]. The framework evaluates the technical support provided by tools with respect to the five levels of social interaction [MC94]: awareness, communication, coordination, group decision making and team-building. By applying the framework to the wiki collaborative features of a wiki tool [GRS12] and investigating their support to each of the interaction aspects, we found that each of the features contributes to support one or more of the social interaction aspects and that all the five levels are actually covered by these features: • Awareness ← Revision history, Watchlist, Notification mechanism, Multi-mode ac- cess to content, (Shared workspace); • Communication ← Multi-mode access to content, Discussion mechanism, (Shared workspace); • Coordination ← Revision history, Watchlist, Notification mechanism; • Decision Making ← Discussion mechanism, Notification mechanism; • Team Building ← Multi-mode access to content, (Shared workspace). Jan Mendling and Stefanie Rinderle-Ma, eds.: Proceedings of EMISA 2016, Gesellschaft für Informatik, Bonn 2016 Empirical Evaluation. For a rigorous evaluation of the impact of the wiki collabora- tive features on the collaborative ontology authoring, we carefully designed the empirical evaluation. Specifically, we focused on the evaluation of (i) the effectiveness of the wiki collaborative features in the process of ontology modelling; (ii) the impact of collaborative features on the collaborative ontology modelling and on the (ontology) entity lifecyle. For each of these aspects we further refined the analysis by defining specific research ques- tions. In detail, for the first aspect we looked at the effectiveness of the features based on three different factors, i.e., DEs involvement (RQ1), required effort (RQ2) and designers’ perception (RQ3). Concerning the impact of the collaborative features, we separately fo- cused on the ontology modelling process (RQ4) and on the entity lifecycle process (RQ5). The empirical study carried out for answering the research questions involved four teams, each composed of a DE and two KEs. Each team was asked to collaboratively model on- tologies for two domains from the pedagogical field. To this end, all the teams members were provided with a modelling tool [GRS12], as well as with chat and email support. Specifically, each team was asked to model the ontology of one of the domains with the modelling tool enhanced with the wiki collaborative features (CMT- Collaborative Mod- elling Tool), and the ontology related to the other domain with the same modelling tool in which the wiki collaborative features were removed (NCMT- Non-Collaborative Mod- elling Tool). The four teams carried out the modelling task on the two domains in two laboratory sessions, one with CMT and the other with NCMT according to a balanced design [Wo00]. Moreover, each session was divided in 5 different phases: the first four phases aimed at simulating an asynchronous collaboration, i.e., DEs and the KE work- ing in different (alternate) phases; the last phase was instead focused on the synchronous collaboration, i.e., all the three team members working together. From the analysis of the results we found that the wiki collaborative features actually in- crease the involvement of DEs in the ontology editing (RQ1). Indeed, when CMT is used, the DEs average number of edited axioms and of editing operations is statistically signifi- cantly greater than when NCMT is used. An inverse pattern occurs instead for KEs, i.e., the KEs average number of edited axioms and of editing operations with CMT is statistically significantly lower than with CMT. Concerning RQ2, the results confirm that the wiki col- laborative features reduce the effort required by team members to communicate. Indeed, the length of the (chat and email) conversations carried out with CMT is statistically sig- nificantly lower than with NCMT. Overall modellers showed a (statistically significant) positive evaluation about the perceived overall effectiveness and ease of use of the wiki collaborative functionalities, while they were a bit more skeptical with respect to the ef- fectiveness of the features in addressing some of the five interaction levels, i.e., support for coordination and team building (RQ3). The obtained results also show that the collabora- tive functionalities have an impact on the ontology modelling process (RQ4). Specifically, it comes out that the process of ontology authoring carried out with CMT is less rigid and encourages more agile interactions among team members than the one carried out with NCMT. By further inspecting this result, we were able to (statistically) confirm that a high usage of the wiki collaborative functionalities correspond to a high dynamism in the ty- pology of activities carried out. Finally, also the created ontology entity lifecycle seems to be affected by the usage of the collaborative functionalities (RQ5). Qualitative differences between the lifecycle of CMT and NCMT ontology entities seems to reveal that CMT en- tities are built according to a process more structured than the one used for NCMT entities, thus suggesting a possibly higher quality of the resulting CMT entities. References [ADR06] Auer, S.; Dietzold, S.; Riechert, T.: OntoWiki - A Tool for Social, Semantic Collabora- tion. In: Proc. of the 5th Int. Semantic Web Conf. volume 4273. Springer, pp. 736–749, 2006. [DGR12] Di Francescomarino, Chiara; Ghidini, Chiara; Rospocher, Marco: Evaluating wiki- enhanced ontology authoring. In: The 18th International Conference on Knowledge Engineering and Knowledge Management (EKAW2012), Galway, Ireland. 2012. [DGR14] Di Francescomarino, Chiara; Ghidini, Chiara; Rospocher, Marco: Evaluating Wiki Collaborative Features in Ontology Authoring. IEEE Trans. Knowl. Data Eng., 26(12):2997–3011, 2014. [Di08] Dimitrova, Vania; Denaux, Ronald; Hart, Glen; Dolbear, Catherine; Holt, Ian; Cohn, Anthony G.: Involving Domain Experts in Authoring OWL Ontologies. In: 7th Int. Semantic Web Conference (ISWC 2008). Springer, 2008. [GRS12] Ghidini, Chiara; Rospocher, Marco; Serafini, Luciano: Modeling in a Wiki with MoKi: Reference Architecture, Implementation, and Usages. International Journal On Ad- vances in Life Sciences, 4(4):111–124, 2012. [MC94] Malone, Thomas W.; Crowston, Kevin: The interdisciplinary study of coordination. ACM Computing Surveys, 26:87–119, 1994. [MRW12] Mendling, Jan; Recker, Jan C.; Wolf, Johannes: Collaboration features in current BPM tools. EMISA Forum, 32(1):48–65, 2012. [Ro14] Rospocher, Marco; Cardillo, Elena; Donadello, Ivan; Serafini, Luciano: On the Collabo- rative Development of Application Ontologies: A Practical Case Study with a SME. In: Knowledge Engineering and Knowledge Management - Proceedings of the 19th Interna- tional Conference, EKAW 2014, Linköping, Sweden, November 24-28, 2014. volume 8876 of LNCS. Springer International Publishing, pp. 469–484, 2014. [RTM14] Rospocher, Marco; Tudorache, Tania; Musen, Mark A.: Investigating Collaboration Dy- namics in Different Ontology Development Environments. In: Knowledge Science, En- gineering and Management - 7th International Conference, KSEM 2014, Sibiu, Roma- nia, October 16-18, 2014. Proceedings. volume 8793. Springer, pp. 302–313, 2014. [Su02] Sure, York; Erdmann, Michael; Angele, Jürgen; Staab, Steffen; Studer, Rudi; Wenke, Dirk: OntoEdit: Collaborative Ontology Development for the Semantic Web. In: 1st Int. Semantic Web Conf. pp. 221–235, Springer, 2002. [Tu10] Tudorache, Tania; Falconer, Sean M.; Noy, Natalya Fridman; Nyulas, Csongor; stn, Tev- fik Bedirhan; Storey, Margaret-Anne D.; Musen, Mark A.: Ontology Development for the Masses: Creating ICD-11 in WebProtégé. In: 17th Int. Conf. EKAW 2010. volume 6317 of LNCS. Springer, pp. 74–89, 2010. [Wo00] Wohlin, Claes; Runeson, Per; Höst, Martin; Ohlsson, Magnus C.; Regnell, Bjöorn; Wesslén, Anders: Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell, MA, USA, 2000.