Collaborative ontology engineering enables the creation of large complex ontologies. However, few projects successfully perform such multi-user ontology modeling. This paper gives an overview of ten di erent ontology engineering projects' infrastructure, architecture and work ows. It especially focuses on issues regarding collaborative ontology modeling. The survey leads on to a discussion of the relative advantages and disadvantages of asynchronous and synchronous modalities of multiuser editing. This discussion highlights issues, trends and problems in the eld of multi-user ontology development.
Semantic technologies give rise to new opportunities in many scienti c disciplines
where complex knowledge and processes need to be recorded. Organizations are
increasingly taking advantage of semantic technologies for modeling this
knowledge. Description logic based knowledge representation languages such a OWL
make such modeling possible. Skilled ontology engineers can create ontological
models of the real world and use tools to check their consistency, as well as
infer information that is only implicitly contained in the model; they thereby can
serendipitously discover new knowledge [
As these ontologies grow larger they expand beyond the capabilities of a single person. A team of multiple ontology engineers is required to build and maintain such large ontologies. However, there are only few modeling projects that manage to successfully perform multi-user ontology development. This paper discusses some of these projects and the design choices and trade-o s they faced. This paper also o ers suggestions for improved editing tools and editing methodologies.
Our survey methodology involved conducting detailed interviews with members of ten di erent ontology engineering projects. Interviews were conducted either by phone or in person. The ontologists were asked to describe their organization and modeling process in detail. They were also asked to complete a questionnaire and encouraged to elaborate verbally on their answers.
The majority of the projects questioned in the user study perform only very basic multi-user ontology development. That is, they allow only one person to edit the ontology at any one time. Some projects edit ontologies asynchronously, while others synchronously work on a single shared knowledge base. A discussion of the relative advantages and disadvantages of these approaches follows.
There are many di erent models for changing traditional structures like databases, computer programs and text documents. Ontologies di er from these traditional document types. Ontologies are used to represent knowledge using description logic. This allows the meaning of statements in an ontology to be expressed in such a way that a computer can understand and make inferences about them. Some traditional systems also capture semantics at a schematic level. These systems do not, however, usually allow frequent and/or dynamic modi cation of these schemata.
Speci cally, ontologies written in the OWL description logic [
The ontology is divided into 2738 modules organized across three monohierarchies (trees) that con ate both compile-time dependencies and editorial groupings. Any number of such modules could be locked and checked-out by any of the twenty members of the multi-national development team. Some modellers would check out modules for many weeks.
Locks created some degree of safety between the modelers, as well as allowing individual modelers to develop from an infrequently changing baseline. However,
occasionally there were problems with two editors independently creating the same concept and/or referencing concepts that did not yet exist.
New releases of the ontology were published roughly every six months. Construction of a new release required all modelers to return and unlock all their locked modules. The complete ontology was then processed by a single curator who iteratively performed tasks such as correcting syntax errors, classifying the complete ontology, correcting any violations of asserted invariants and removing reported redundancy until the ontology classi ed without error. The episodic curation process typically took one to two weeks, in part because classi cation alone was a lengthy (2-3 hour) process. Any further modeling work done during this curation time needed to be repeated after the completion of the new release. Ordnance Survey 2 The Ordnance Survey collects and maintains very detailed topographic data for Great Britain. It is building an ontology of relationships between topographic features. Of the three people in the organization who routinely edit ontologies, only one engineer is allowed to make changes to this ontology at any one time. Changes are discussed in face-to-face meetings or via email.
The APSF-DDTF is using a simpli ed cut-down version of the Protege OWL
ontology editor [
Additionally, on rare occasions, some multi-user concurrent editing is done. Independent parts of the ontology are edited concurrently. The changes are then manually stitched together into a single ontology using a XML editing tool. BioPax Consortium 4 The BioPax Consortium is a world-wide e ort to develop a multi-level ontology of biological pathway information. The ontology le is emailed between members for editing. Only one person changes the le at any one time. Changes are discussed in conference calls.
U.S. National Cancer Institute 5 The National Cancer Institute in the United States is continually re ning the NCI-Thesuarus, a large biomedical ontology of diseases, therapies and other topics related to cancer research.
The ontology is currently being maintained using a set of tools from the Apelon corporation. A group of 16 editors work on the ontology. These editors import a copy of the ontology into their own private individual database and make changes independent of each other. Every month all changes are sent to a central curator who integrates the changes and resolves any con icts. The curator 2 http://www.ordnancesurvey.co.uk 3 http://www.apsf.org 4 http://www.biopax.org 5 http://www.cancer.gov works for three to eight hours to produce a new updated baseline version of the knowledge base. This baseline is then re-imported by all the editors and they again start working independently.
The con ict resolution step is a bottleneck in the NCI process. While the curator is working to produce a new baseline all the other editors are idle. Editors may not make any changes during that time, since these would be lost with the release of the new baseline.
The NCI must balance the loss of productivity while a new baseline is created with the danger of two editors independently creating concepts of the same name, or worse: concepts with di erent names that were meant to express the same thing. Ideally, infrequently creating new baselines minimizes the time lost. However, frequently creating new baselines minimizes the possibility of problems due to overlapping work.
Another important issue for the NCI is the lack of integration between tools: the work ow, con ict resolution, editing and publishing tools are separate and must be manually integrated. For example: editors must manually email their change set to the curator for integration.
In order to address its modeling di culties the NCI is planning to switch to a di erent editing paradigm. They want to host their ontology on a central database server to which individual clients connect using Protege OWL. Any changes immediately show up for all connected editors. If two modelers start editing the same concept at the same time, one modeler's changes will be lost when the other commits.
to collaboratively build an ontology for biological and medical investigations (formally known as the Functional Genomics Investigation Ontology (FuGO) project).
The project currently includes 13 specialized communities with a total of 70 developers collaborating across four di erent countries. Only one person may access the ontology at any one time. Major changes are made in face-to-face meetings every six months with a single scribe editing. Minor changes require approval from everyone in the community to make sure that no one else is already working on a change. Since the community is world-wide, time di erences between countries result in a two day delay for even the most minor change (e.g. adding a comment).
The project uses Protege OWL to edit an ontology stored in a subversion repository. Side projects are sometimes launched as branches from the main ontology. These must be manually merged into the main ontology once complete. Imperial Cancer Research 7 Imperial Cancer Research at University College London is building an OWL ontology of diseases and treatments related to breast cancer. Existing ontologies were found to be too large and confusing to use. The ontology is being maintained by a single person. 6 http://obi.sourceforge.net 7 http://www.cancerresearchuk.org
Terminology) project was a large e ort to create very large reference terminology of medicine. At its peak, the project employed thirty domain experts around the USA to concurrently construct the ontology. It was completed in the year 2001.
The SNOMED RT project endeavored to create a very high-quality
knowledge base by using dual independent review. Sets of concepts are assigned and
distributed out to modelers. Each concept is sent to at least two di erent
modelers. These modelers create de nitions for their assigned concepts and then send
them back into a central server. An automated con ict resolution tool analyses
the resulting de nitions and, if the two modelers de nitions don't match,
initiates a con ict resolution procedure. The con ict resolution tool allows modelers
to discuss and eventually reach a consensus on how to best model the term in
question. 2 to 6% of concepts usually result in con icts, though this is very much
dependent on the nature of the subject area being modeled. Some areas produce
no con icts, while in others almost every concept results in a con ict [
SNOMED CT (Clinical Terms) is continuing the development of SNOMED merged with the CTV3 terminology (formally known as Read Codes) used in the United Kingdom. Contrary to the RT development, CT no longer uses dual independent review. The ontology is primarily edited synchronously in a central location (Chicago) with all modelers connecting to a central database. Additionally, three or four modelers/centers outside of the main location asynchronously edit branches of the terminology. These branches are integrated into the main ontology approximately every two weeks. While this integration is going on all asynchronous editing stops, but it is usually performed overnight, so little actual downtime occurs. Any con icts that arise are assigned to and corrected by modelers in the central location. The integration results in a new baseline ontology which is distributed to the outside modelers for reviewed asynchronous editing.
Editing is performed using Apelon's Terminology Development Environment (TDE).
Clinical e-Science Framework 9 The CLEF project at the University of Manchester and University College London aims to integrate, organize and extract data from clinical information (e.g. medical records).
The two researchers at the University of Manchester have collaboratively constructed an ontology of medical concepts for use in the project. Only one person makes changes to the ontology at any one time. Changes are discussed face-to-face.
CancerGrid 10 CancerGrid is a consortium of ontology and software developers that is working on providing terminology servers to cancer research centers throughout the UK. They integrate various large ontologies (GO, SNOMED, NCI-Thesaurus) into each server and allow researchers to access, query, annotate and locally extend the uni ed knowledge corpus.
While CancerGrid is not directly developing an ontology of its own, it is very much involved in multi-user ontology engineering and provided us with requirements for future tools in light of its aims. 3.2
1. Asynchronous editing: multiple ontology engineers working on individual copies of a knowledge base, submitting their changes to a central server. 2. Synchronous editing: multiple ontology engineers symultaniously connected to a single knowledge base on a server with changes taking e ect immediately.
Synchronous editing has the advantage that changes (and errors) are
immediately visible to all parties involved. Another advantage is that there is only
a single master version of the ontology at any one time. With asynchronous
editing, there is always the potential of multiple branches [
However, in a synchronous editing environment all parties must always be connected to the same server. Disconnected editing on trains and airplanes is not possible. This also prevents editors from going away and working in a sandbox environment to test a potentially harmful series of changes. In an asynchronous environment potentially dangerous changes can be independently developed, veri ed to be safe and then integrated with the rest of the ontology.
A mistake, con ict, or inconsistency can be disastrous in a synchronous environment. An error could result in the knowledge base becoming unusable for all connected users and editors. If the integrity of the knowledge base is critical then a synchronous environment is particularly dangerous. In contrast, in an asynchronous environment an error can be undone before it a ects anyone else, thereby reducing the cost associated with xing errors. Also, even if an error damages the complete knowledge base, its state can be selectively rolled back with relative ease; while, in a synchronous environment, rolling back the knowledge base to a previous state (or snapshot) will erase all the work done up to that point (just not the o ending update), and stop everyone's progress while the roll-back is being executed.
The asynchronous work ow requires that changes be integrated into the complete knowledge base as editors submit them. This process is more complex than clients directly operating on the central ontology. Di erent lengths of check-outs create a degree of uncertainty about the state of the knowledge base. A modeler could at any time submit a signi cant change. Most projects using asynchronous editing therefore every so often collate all changes into a build/baseline/release version, rather than being in a state of continuous ux. 4.2
Means of preventing con icts can also be divided into two: 1. Locking: a section of the knowledge base is locked in order to prevent multiple people editing it at the same time. 2. Optimistic non-locking: locks are not used. The inevitable con icts are xed after they occur.
The advantage of locking is that it pro-actively prevents con icts. However,
editing becomes more di cult, especially as the number of locks increases and
other users are prevented from accessing large portions of the knowledge base.
Locking (in an asynchronous setting) also requires some means of modularizing
or segmenting the ontology [
Project management is more di cult using non-locking methods. The ability for anyone to edit anything at any time makes it di cult for modelers to be aware of each other's editing. Locks force users to edit their pre-declared or assigned areas.
In the absence of locks non-unique de nition con icts [
The majority of ontology engineers questioned in our survey preferred an asynchronous approach to multi-user ontology engineering to a synchronous methodology (see Figure 2). Anecdotal evidence suggests that more technical ontology authors with backgrounds in software engineering tended to prefer the asynchronous model, while ontologists with life-science specializations, without backgrounds in computer science tended to prefer a synchronous model.
The more technical ontologists tended to be comfortable with asynchronous CVS-style systems and were aware of overlapping edit problems in a synchronous system. Less technical ontologists were unaware of potential problems and wanted a multi-user ontology editing to be as simple as possible. They regarded an asynchronous editing system as too complicated. 5
Statistics of the various project were compiled. These are as follows: Ontology structure The ontologies under development have a median of 3000 classes (mean of 58380), with an estimated average of about 3 restrictions per class and a maximum number of about 25 restrictions on a single class. 60% of ontologies under development are single large ontology, with only 20% of projects using mutually importing ontology modules and another 20% creating multiple disconnected ontologies (see Figure 4).
Ontology criticality Most of the ontologies under development are not being used in critical systems within the organization. Errors mostly cause only minor inconvenience (see Figure 3).
Ontology language The majority of ontologies are being developed using
Protege OWL [
Communication Email is the preferred means of communication for almost all organizations questioned. Only SNOMED and GALEN have dedicated work ow, con ict resolution and communication tools. Other communication technologies include: shared desktop clients, face-to-face meetings and the telephone. All of these are used to approximately the same degree, but none as universally as email. 6
Multi-user ontology engineering is not very widely practiced. This section analyses why this is the case and what can be done to enable better concurrent ontology development.
A xed questionnaire was used in our evaluation. Project representatives were encouraged to elaborate on their answers and/or suggest new alternative answers. However, beyond several adjustments resulting from a prototype questionnaire presented to two local projects (Clef and OpenGALEN), no additional categories were suggested. Interviews were conducted either by phone or inperson with one representative from each project. 6.1
Figure 7 shows the main problems in multi-user ontology editing. Most projects who's members we interviewed are distributed over multiple physical locations, many are even international (see section 5). This inevitably makes co-ordinating ontology modeling di cult. Keeping control of the editing process seems to be the main problem developers faced when attempting to do multi-user ontology development. Another major source of problems included unintended changes which occurred due to the concurrent nature of the editing process.
The most popular solution to concurrent editing problems is abandoning the idea altogether. As also mentioned in section 3.2, Figure 8 shows that most projects resort to passing their ontology between single individuals for single-user editing. This seems to be a stopgap measure to allow developers to at least make some progress. The general consensus is that ontology engineers would certainly prefer a true multi-user work ow, if it was available. 6.3
Fig. 9. What currently inhibits multi-user Fig. 10. Most important characteristics of ontology development? (x-axis: number of a multi-user ontology editing system (xprojects seeing each inhibiting issue) axis: number of projects seeing each characteristic as desirable)
Figure 9 shows what is inhibiting organizations from performing their desired form of multi-user ontology editing. Having su cient tool support is clearly critical for concurrent editing, with trained ontology engineers (and training material) also being very important.
Concurrent editing of shared ontologies is both di cult and error prone. Good multi-user editing tools and quality training literature does not exist, so ontology engineers resort to makeshift solutions such single user editing. 7
This section analyses the requirements for tools to support multi-user ontology editing.
The most important tool in building ontologies is clearly the ontology editor.
Popular ontology editors include: Protege OWL [
To our knowledge Protege OWL o ers some support for synchronous development and Apelon supports some asynchronous development. Still, both solutions could be signi cantly improved. For example, as far as we are aware, none of the above tools support any kind of locking.
A multi-user tool requires a di erent set of features than a single-user application. Figure 10 shows which characteristics users in the questioned organizations considered most important for such a tool.
Unsurprisingly, simplicity was very important for the ontology engineers. They found current applications too complicated to use. However, these developers were equally concerned about provenance information. They wished that any multi-user tool automatically records as much provenance data as possible.
A low granularity of edits (the ability to select or edit a very small section of an ontology for a short time), was not seen as very important. This was probably due to ontologists not seeing the connection between this feature and the ability to achieve a high degree of concurrency, which was classed as an important feature. The lower to granularly of each edit, the more concurrent editors the tool can support.
Surprisingly, users also found error prevention, safety and work ow support to be relatively unimportant. These were perceived to add unnecessary complexity to the editing process. Users commented that they would rather have the ability to diagnose errors and rollback erroneous changes than be constrained by a system that imposes signi cant restrictions to ensure total safety when editing. Also, rather than be constrained by a xed work ow, ontology engineers preferred a greater degree of exibility. 11 http://www.topbraidcomposer.com 12 http://www.apelon.com 13 http://www.altova.com/features_owl.html 7.3
Asynchronous edit duration In an asynchronous editing scenario the length of time of a check-out is an important design consideration. The longer an ontology engineer independently works on a section of the ontology, the greater the chance for con ict. This is especially likely in an optimistic non-locking scenario. The greater the number of con icts, the more time must be spent by a curator (potentially in single-user mode, see section 6.2) or by a con ict resolution system to negotiate a solution to the con icts.
With that in mind, a longer duration of edit/check-outs should necessitate a system of stricter locks to prevent a greater number of con icts. However, stricter locks also increase the barriers to editing (see section 4.2). This ability to carry out widespread edits is especially important with long check-out durations, since an ontology engineer might nd that he or she needs to edit something that was recently locked by someone else and therefore is likely to remain locked for some time to come. So, in a long check-out scenario there is a con ict between con ict prevention and widespread editing. A system designer must decide if widespread editing or con ict prevention is the priority and choose to implement either non-locking or locking respectively.
This trade o is especially relevant, since we found that the majority of ontology engineers anticipated that, if working in an asynchronous ontology development system, their duration of edit would range between one to ve days. This is illustrated by Figure 11. Synchronous edit duration In a synchronous editing system multiple editors should be prevented from simultaneously changing the same axiom in an ontology. However, check-outs, as such, do not exist, so there is no need to consider duration of edits with relation to locking, beyond the simple case of overlapping edits. 7.4
When people build ontologies they inevitably make mistakes. If a statement in
an ontology represents a logical contradiction, then a description logic classi er
will highlight the unsatis able concepts. Debugging tools [
Figure 12 shows the frequency of certain categories of errors resulting from multi-user interactions, as well as how di cult these errors are to diagnose and x, as reported by the interviewed ontology engineers.
Errors without any known cause were the most di cult to x, as might be expected, but these kinds of errors were also quite rare. Property statements contradictions were the second most di cult to x. Debugging tools handle these kinds of errors quite well.
Failure to achieve the desired subsumption, while the most common error, was not as di cult to x as the two aforementioned errors. So, correcting a contradiction was more di cult than achieving a desired subsumption. However, while ontology debugging tools help editors diagnose contradictions, they do not help with suggesting how to achieve desired subsumption relationships. That is: it would be good if the ontologist could indicate a desired subsumption relationship and the tool could generate a series of suggestions as to how to achieve this.
Interestingly, no one mentioned unexpected subsumption as even slightly di cult to x, even though it was the third most encountered error. This seems to indicate that unexpected subsumption is less an error and more a natural part of ontology development. Simply because a new subsumption relation is unexpected does not mean that it that it is necessarily undesireable, quite the opposite, in fact.
In this paper we have explored the ndings from our survey of ten ontology development projects with relation to multi-user ontology development. We found that most organizations nd multi-user ontology development very di cult, primarily because of a distinct lack easy-to-use tools. Because of this most projects end up resorting to a pass-the-pen mode of editing where only one user may edit the ontology at any one time.
An asynchronous mode of editing using locking seems appropriate for a majority of organizations, based on their requirements, modeling styles and preferences. This style of editing can also aid task coordination, which is the single greatest problem reported during multi-user editing. Other modes of editing are also applicable for more specialized projects with speci c requirements.
Development tools need to be further improved beyond their present state. The survey suggests that debugging and refactoring tools would greatly enhance the multi-user development process, but their existence is relatively unknown. Additionally, a use case for a new tool was found: such a tool, given a desired subsumption relation, would suggest changes that result in the manifestation of this relation.
All these factors point to an overarching need for a multi-user ontology development methodology. That is: a clear guide for how to perform large-scale ontology engineering.