Over the last decade, ontologies have become the mainstay in the biomedical domain. Their size and complexity, as well as, the required expert domain knowledge to create these ontologies have increased significantly. In addition, many projects resort to collaborative approaches for building these ontologies, using the Internet as a cooperation platform. While online collaborative projects have become common, the processes that drive these collaborations are still not well understood. In this paper, we are investigating novel approaches and visualizations using Markov chains to improve our understanding of how users build large, real-world ontologies. Using our novel methods, we analyze, visualize and compare the editing behavior of users in two collaborative ontologyengineering projects from the biomedical domain. The contributions of our work are two-fold. First, we visually explore the editing behavior and dynamics of users in collaborative ontology-engineering projects; and second, we quantify the differences between the editing behaviors in these two projects. We also discuss the implications and potential applications of our findings, which, we believe, may be used to create adaptive user interfaces that better support the editing behaviors of the users.
new requirements for the ontology-engineering process are emerging. In order to better support users in editing these large real-world ontologies collaboratively, and to support the project managers to get an overview of the entire process, we need to better understand the intricacies of the collaborative process for building ontologies.
Uncovering such new insights—which can potentially be used to adapt and improve existing ontology-engineering tools, or devise new and expand existing development and evaluation strategies—represents a very important first step towards overall improved and more easily maintainable structured knowledge representations.
In this paper, we focus on visually analyzing, exploring and comparing user
editing behavior across two different collaborative ontology-engineering projects from the
biomedical domain. To that end, we expand on our previous analyses [
The remainder of the paper is structured as follows: We discuss the related work (Section 2), then we describe the datasets and methods to extract user-editing behaviors from the change history of collaborative ontology-engineering projects (Section 3). We present the results of our analyses in Section 4, and then discuss the implications and potential applications of the presented visualizations in Section 5. 2
In general, research in the field of ontology engineering addresses all the tasks, actions,
tools and processes required for developing and evaluating ontologies [
To learn more about the impact and implications of collaboration and consensus
finding in collaborative ontology-engineering projects, researchers have analyzed
several aspects of such projects. For example, Falconer et al. [
In 2014, Van Laere et al. [
To support the collaborative development of ontologies, the Semantic Web
community has already developed a number of tools and visualizations. For example, Prote´ge´,
and its versions for collaborative ontology development, such as WebProte´ge´ [
Many of the existing ontology-engineering tools already provide built-in
functionality or plug-ins to visualize and browse ontologies [
In our previous work, we have used Markov chains to visualize different aspects
of the editing behavior of users in collaborative ontology-engineering projects [
We used the change logs of two real-world ontology-engineering projects to conduct
the analyses presented in this paper. Both projects were developed with iCAT [
ICTM 1, 506 67, 522
27 2011/02/02 2013/07/17 2.5 years by each user. Each entry in the record stores additional metadata about the change, such as the user who performed the change, a brief description of the change, the timestamp, and all the classes and properties involved in the change. Additionally, we have removed all automatically executed changes.
Using these additional metadata, we can create sequences of change actions for each user, which can then be used to fit a (first-order) Markov chain model. A brief characterization of the datasets can be found below and in Table 1. Note that we have removed users with less than 2 changes from our analysis as no transitions between changes could be observed.
Organization (WHO), is the international standard for diagnostic classification used to
encode information relevant to epidemiology, health management, and clinical use in
nearly all of the United Nations’ 193 member states. The 11th revision of the
classification, ICD-11,3 is currently in progress, with a planned finalization in 2018. In contrast
to previous revisions, ICD-11 is developed as a rich OWL ontology [
The International Classification of Traditional Medicine (ICTM)4 is a project that aimed to produce an international standard terminology and classification for diagnoses and interventions in Traditional Medicine and was led by WHO. Analogously to ICD11, ICTM was developed collaboratively as an OWL ontology with the goal of unifying the traditional medicine practices from China, Japan and Korea. The content of the ontology is authored in English, Chinese, Japanese and Korean. More than 20 domain experts from the three countries developed ICTM, using a customized version of WebProte´ge´, until the development of ICTM ended in 2012. Currently, there is ongoing work to make ICTM a chapter in ICD-11.
3 http://who.int/classifications/icd/ICDRevision/ 4 http://who.int/mediacentre/news/notes/2010/trad_medicine_ 20101207/en/ 3.2
In order to analyze the editing behavior of users in two collaborative ontology-engineering projects, we have extracted and aggregated several different types of changes (see Table 2). Each change-type represents one action that a user performed in the user interface. For example, Edit Property Value describes a change, where an existing value of a property of an entity is edited by a user. In contrast, Add Property Value describes the action of adding a value to a property of an entity, which was previously empty. We extracted these change types from the change history recorded by iCAT. Note that even though ICD-11 and ICTM were both created using iCAT, not all types of changes were performed in both projects during our observation periods.
By extracting and analyzing the sequences of change-actions of all users in ICD11 and ICTM, we can calculate the “average” editing behavior of all users involved in the corresponding project—in the form of a first-order Markov chain—and identify commonly followed workflows. Further, whenever more than 30 minutes have passed between two consecutive changes of one user, we have added a BREAK state. This allows us, not only to learn more about common workflows among users, but also which actions are conducted before users take a break, and which actions are conducted, once they resume work.
It is important to understand that the types of changes listed in Table 2 are manually aggregated change-types of all the different types of changes created by iCAT. The visualizations and analyses are not limited to these, and could be easily applied to a different set of change-logs created by a different ontology-development tool that provides similarly granular log-information. Aggregating similar change types into one action helps make the visualizations easier to read. 3.3
A Markov chain consists of a finite state-space S, where each state s1, s2, ..., sn ∈ S with n = |S| and a transition matrix P , which lists all probabilities pij to traverse from the state si to the state sj , and for each i, Pj pij = 1.
For example, if the state space of one specific Markov chain consists of all the types of changes that can be conducted on an ontology, the values listed in the transition matrix P reflect the probabilities of a user to conduct a specific type of change immediately after that user conducted the same or a different type of change.
Similarly, Markov chains can be used to model the transition probabilities between multiple transitions or higher orders. This means that the next state does not only depend on the current state, but on a sequence of k previous states, as well.
In our previous work, we have already demonstrated that higher-order Markov chain
models can be extracted from the logs of changes of different collaborative
ontologyengineering projects [
To be able to calculate the transition matrix P , we first have to define W with each element wij representing the number of transitions between states si and sj . Hence, Pj wij describes the absolute number of occurrences of state si in the corresponding dataset. Finally, we calculate P by normalizing each row of W by its corresponding ℓ1-norm, and use it for our visualizations.
To be able to compare two projects, we calculate the transition matrix Qabs by subtracting WICT M from WICD−11, and again normalizing each row by its ℓ1-norm, as described in Equation 1.
Qabs = ||WICD−11 − WICT M ||1 (1)
Note that the transition probabilities in Qabs reflect the differences between the absolute numbers of occurrences of change-type transitions of the two datasets. After normalization, frequent transitions, which are mostly appearing in only one of the two dataset, will either be close to 1.0 for the minuend (WICD−11) or −1.0 for the subtrahend (WICT M ).
In contrast, Qrel = PICD − PICT M represents the difference between the relative transition probabilities of the change-type transitions. Analogously, after normalization, transition probabilities will range between 1.0 and −1.0. Hence, we can either decide to compare the absolute number of transitions between the different change-type actions (Qabs), or the relative transition probabilities (Qrel).
A detailed description of all the steps necessary to apply Markov chains on
collaborative ontology-engineering projects, as well as additional results that complement the
analysis presented in this paper can be found in Walk et al. [
a S h
Additionally, we can observe that users who delete a property value are equally likely to delete another property value or add a new property value to a class. For both projects, we can further see that users, who conducted a Remove Superclass change, are very likely going to create a new class next or move classes in the hierarchy (Create Class and Move Class(es)). However, after moving a class, users in ICD-11 exhibit a higher tendency to edit a property value, while users in ICTM are likelier to remove another superclass.
For both projects, there does not appear to be a specific change action that users conduct before taking a break from work (see columns of BREAK in Figure 1), and the probabilities to conduct specific types of changes when returning from a break are according to the frequencies of the different types of changes depicted in the histograms (see rows of BREAK in Figure 1).
To further highlight the differences between the editing behaviors of the users of the two projects, we visualized Qabs (see Figure 2(a)) and Qrel (see Figure 2(b)). As ICD11 exhibits roughly six times the amount of changes compared to ICTM (see Table 1), transition probabilities in Qabs are mostly positive. The only exceptions are Create Reference and Import Property, which were performed more often by the users of ICTM. As described before, we can now see that users in ICTM have a higher tendency to first conduct a Remove Superclass change, immediately followed by a Move Class(es) change than users who work on ICD-11. In contrast, users in ICD-11 are likelier to conduct a Create Class change after removing a superclass (Remove Superclass).
When looking at Qrel, this difference becomes even stronger (Figure 2(b)). We can also see that specific workflows, such as creating a class, editing a property value and then adding a property value are more dominant for ICTM, which is likely caused by the overall lower number of changes and thus, their increased relative importance. In contrast, users in ICD-11 exhibit a higher tendency than users of ICTM to consecutively perform the same change-type actions, as depicted by the diagonal in Figures 2(a) and 2(b). 5
The analyses of the presented visualizations provide important insights into the editing behavior of users in collaborative ontology-engineering projects, which can not be inferred directly from static data. We have demonstrated that common workflows can be identified when exploring the editing behavior visualizations presented in Section 4. We believe that this newly obtained information could be used by ontology-engineering tool developers to identify opportunities for improving the ontology-development tool. For example, the user interface of the tool can be better aligned to the observed workflows, and tool developers could use the information about the obtained sequences of changes to adapt the interface with the goal of reducing the number of clicks that are required to perform certain workflows. te g f th r
o u n o r l lsob tiid sno f)o ceo
E it A m th )a re ac tto re ( s e o t
U p b h
g f ty ( i o e l
e t e n g ta h
t
Further, the presented visualizations could act as source of information for automatically adapting interfaces, which guide users through the process of developing an ontology by considering their historical editing behavior. We have also observed that users tend to edit “vertically” in the ontology (e.g., if they edit the property value of a class, they will do so for all the subclasses). Such a workflow suggests that a tabular editing interface, similar to spreadsheets, presenting one class per row, would speed up the content entry by the users.
The visualization of the differences in editing behaviors between the two projects may be used by project administrators to identify and analyze differences in the workflows between projects, and especially between the ones that use different tools or guidelines for creating the ontology. In contrast to existing visualizations, our presented method specifically concentrates on the sequential nature of the change-log data, and provides novel insights into the dynamic nature of collaborative ontology-engineering projects.
For future work, we plan on further refining the presented analyses to compare the editing behavior of different groups of users. For example, biologists might exhibit different workflows and patterns, when developing an ontology than chemists or computer scientists do. To extend the utility of the Markov chain visualization, we plan on making the visualizations interactive. This extension would allow us to dynamically aggregate similar types of changes into abstract classes of changes, which could be expanded and collapsed by users while exploring the visualization. Additionally, this extension would make it possible to visualize higher-order Markov chains, and avoid visual clutter due to the increased number of states. Finally, we are also interested in comparing the editing behavior of users across different ontology-development tools to assert the extent to which the tool influences the editing behavior. 6
In this paper we have extended the analyses from our previous work, and demonstrated how to visualize and compare the editing behavior of users of two different collaborative ontology-engineering projects from the biomedical domain. In that process, we have uncovered and discussed several editing workflows, and we have presented a novel visualization, which highlights the differences in editing behaviors between two projects. Finally, we have discussed the implications of our findings for ontology tool developers and project administrators, and we have outlined potential applications of the visualizations for future work.
Acknowledgments This work was supported by Grant GM086587 from the U.S. National Institute of General Medical Sciences (NIGMS) of the National Institutes of Health. The Prote´ge´ project is support by NIGMS grant GM103316.