Ontology maturing as a conceptual process model is based on the assumption that ontology engineering is a continuous collaborative and informal learning process and always embedded in tasks that make use of the ontology to be developed. For supporting ontology maturing, we need lightweight and easy-to-use tools integrating usage and construction processes of ontologies. Within two applications - ImageNotion for semantic annotation of images and SOBOLEO for semantically enriched social bookmarking - we have shown that such ontology maturing support is feasible with the help of Web 2.0 technologies. In this paper, we want to present the conclusions from two evaluation sessions with end users and summarize requirements for further development.
The first wave of semantic (web) applications has shown that ontologies are well-suited
for sophisticated ways of retrieval of relevant resources, but traditional ontology
engineering methodologies and tools suffer from the underlying assumption that a few
modelling experts have to create an ontology for many users. In order to keep the
ontology in line with the intended usage, cumbersome procedures are introduced that lead
to delayed and often error-prone updates to the ontology (cf. [
Starting from the insight that building an ontology is essentially formalizing an
understanding of a particular domain, we conceive ontology engineering as a continuous
collaborative learning process, which we call ontology maturing [
In this paper, we want to present the results of a formative evaluation of these tools with end users and the conclusions for future developments. In section 2, we first briefly present the ontology maturing process model before we sketch the tools and their functionality in section 3. In section 4, we present the results from the evaluation sessions and the conclusions for future enhancements. 2
Starting point of our ontology maturing process model were the shortcomings of the
usual separation of creation and usage processes [
This led us to rethink ontology engineering as a collaborative and work-integrated
activity. In this view, users themselves (within, e.g., communities of practice) can modify
the underlying ontology of a semantic application, e.g., add new ontology elements or
modify existing ones. This new perspective, motivated by constructivist views on
learning (see also [
In particular for emerging ideas and concepts, it is not possible to directly integrate them into an ontology as they are not clearly defined, yet. That means, the development of an ontology underlies a process of continuous evolution where different levels of formality might co-exist within one ontology. The outcome is an adequate level of formality in the ontology, avoiding both overformalization and the inability to apply semantic algorithms. 2.2
To operationalize this view, we have developed the ontology maturing process model that structures the ontology engineering process into four phases (see Fig. 1): 1. Emergence of ideas. New ideas emerge and are introduced by individuals as new concept ideas or informal tags. These are ad-hoc and not well-defined, rather descriptive, e.g. with a text label. They are individually used and informally communicated. 2. Consolidation in Communities. Through the collaborative (re-)usage of the concept symbols (tags) within the community, a common vocabulary (or folksonomy) develops. The concept ideas are refined, useless or incorrect ones are rejected. The emerging vocabulary, which is shared among the community members, is still without formal semantics. 3. Formalization. Within the third phase, the community begins to organize the concepts into relations. These can be taxonomical (hierarchical) ones as well as arbitrary ad-hoc relations, e.g., in the course of becoming aware of different abstraction levels. This results in lightweight ontologies that rely primarily on inferencing based on subconcept relations. 4. Axiomatization. In the last phase the adding of axioms allows and improves for inferencing processes, e.g. in query answering systems. This step requires a high level of competence in logical formalism so that this phase is usually done with the aid of knowledge engineers.
It is important to note that ontology maturing does not assume that ontologies are built from scratch, but can be equally applied to already existent core ontologies used for community seeding. Likewise, this model must not be misunderstood as a strictly linear process; rather real ontology development processes will consist of various iterations between the four different phases.
We identified semantic annotation and retrieval of resources as one possible use case where the ontology maturing process model can demonstrate its potential. We will concentrate on this use case for the rest of this paper, although other semantic applications, e.g. for expert finding or description of web services could benefit from the usage of the ontology maturing process model as well. 3
Our applications realize the ontology maturing process model by offering work-integrated ontology development and an easy-to-use interface to allow the usage of semantic technologies also for “ordinary” people. SOBOLEO allows for the semantic annotation and retrieval of web resources, the ImageNotion tool for the semantic annotation and retrieval of images. In this section, we will give a brief introdution of these applications.
SOBOLEO [
SOBOLEO (see Fig. 2) consists of four major parts: (1) a collaborative real time editor for changing the ontology, (2) a tool for the annotation of web resources, (3) a semantic search engine for the annotated web resources, and (4) an ontology browser for navigating the ontology and the index of the web resources. The users within one community create and maintain one ontology and one shared index of web resources collaboratively.
Thus, the users can create, extend and maintain ontologies according to the SKOS
Core Vocabulary [
In this way, new concept ideas are seamlessly gathered when occurring (maturing phase 1) and existing ones are refined or corrected (maturing phase 2). The users can structure the concepts with hierarchical relations (broader and narrower) or indicate that they are “related”. These relations are also considered by the semantic search engine. That means, the users can improve the retrieval of their annotated web resources by adding and refining ontology structures (maturing phase 3).
ImageNotion [
One pecularity of communities in the area of semantic image annotation is that we usually have two separate roles and groups of interest: content owners (providing the images) and image users. The content owners use imagenotions for annotation to improve the findability. Image users use imagenotions for searching images they are interested in, e.g. for commercial usage in media. Both of these groups have to collaborate and thus engage in maturing of imagenotions to improve the quality of semantic annotation of images. 4
As ontology maturing support has to follow a participatory philosophy, it was important to have formative evaluation of our prototypes early on. End-users recently evaluated both tools in two different environments and evaluation settings. In the following, we describe their respective evaluation setups and summarize the results. 4.1
We evaluated SOBOLEO in two separate sessions. The first evaluation took place from April 16-30, 2007, within the scope of the Collaborative Knowledge Construction Challenge within the Workshop on Social and Collaborative Construction of Structured Knowledge held at the 16th International World Wide Web Conference1. We provided a basic ontology to facilitate getting started and to give thematical orientation for the participants. This ontology was tailored to the research domain as a whole with concepts like ’research topic‘, ’people‘, ’institution‘, ’publication‘, and ’event‘. Everyone was free to participate and contribute information about their research domain. At the end, they were asked to provide feedback. Altogether, 49 users registered and 33 contributed actively to the challenge.
During this evaluation, the participants added in total 202 new concepts and 393 concept relations to the ontology. Further, they collected 155 web resources, which they annotated with 3 concepts per resource on average. None of the users had the opportunity to meet other users using SOBOLEO at the same time. Thus, the chat functionality was barely used; only for testing.
Summarizing the feedbacks, the participants appreciated the ease-to-use of SOBOLEO and having a shared ontology. They emphasized in particular the editor’s realtime nature. The users further enjoyed the simple way for annotating web resources with concepts or tags, which are then automatically added. Thus, to have the possibility to integrate not yet well defined concepts but somthing like ”starter concepts“ and, in this way, to ”get the ontology building almost for free“. For improving SOBOLEO, the users pointed out several times that they missed a personal view on the data, i.e. on the own annotated resources but also on the ontology (especially in case of a growing and dispersing user base). Although the users appreciated the messages/chat pane informing about changes and for communication with other users, the users expressed the wish to have more possiblitites to discuss and be informed about modification (on ”own“ data) by other users. Thus, to gain more translucence and awareness, especially as they could not experience working together simultaneously. A further aspect was to have better support for identifying or suggesting conflicts, synonymous concepts and broader-narrower relations in order to facilitate the maintenance of the ontology.
The second evaluation of SOBOLEO took place within the scope of the project “Im Wissensnetz”2 (“In the Knowledge Web”). This evaluation was especially intended to test usability (especially goal/task support) of SOBOLEO and was assisted by thinkingaloud techniques and screen recording tools. Within two one-hour sessions, four users had to carry out specific tasks simulating the usage of SOBOLEO within their daily work activities. Half of the users were researchers of the rapid prototyping domain and half of them patent experts for German research. All of them were unexperienced in ontology development. We provided a basic ontology with 31 concepts to start with that was thematically tailored to the rapid prototyping domain.
During the second evaluation, the four users created 6 new concepts. This low number can be traced back to the given tasks, which did not demand the explicit creation of new concepts. Instead the tasks were tailored to gain orientation within the ontology by letting the users place or add synonyms to existing concepts. Thus, the users added 11 synonyms and 21 concept relations. During the annotation specific tasks, they collected in total 42 web resources, which they annotated with 2.5 concepts per resource in average.
The users appreciated SOBOLEO for its easy of use. Some of the users had some problems at the beginning due to their very basic knowledge in ontologies and were confused by the concept editing functionality. But a learning effect could be observed shortly. The chat turned out to be an essentiell utility; especially for simultaneous working. For instance, two users had problems in placing concepts in the given ontology because they had only basic knowledge of the rapid prototyping domain. In consequence, they began to ask their colleagues for help via the integrated chat functionality. Nevertheless, the chat appeared to be too simple. For improvement, the users wished to have a better integration of what is discussed and where the changes are done. Further extended functionalities like chat rooms as well as more documentation to understand how and why decisions and modifications are done (also for later use) were required. This evaluation showed as well that translucence and awareness are crucial factors in collaborative ontology development. 4.2
The first stable version of the ImageNotion tool was evaluated in June 2007 by experienced image annotators and librarians having minimal ontology background within the scope of the IMAGINATION project. Our aim was to evaluate whether they are able to collaboratively create ontologies in combination with the semantic annotation of images using the ImageNotion tool. Six people participated at the workshop. The reference set consisted of 854 images from the preselected domains “world war 1” and “European politicians”. One participant had well-founded background knowledge about semantic formalism; two of the participants (user 2 and 3) had many experiences with tag based annotation systems but no experiences with semantic formalisms and applications. The other three participants were familiar with thesauri, but not with the creation of ontologies or with image annotation systems.
The results of our evaluation were generated in two hours by the partcipants. Comparing the sum of work steps of all users for ontology maturing activities and for annotation activities, table 1 shows that the number of work steps for the work process ontology maturing (115 steps) is higher than the number of worksteps for the semantic image annotation. This shows the need for a work integrated ontology maturing. From the total number of 46 created imagenotions, 26 imagenotions were directly used for the semantic annotation of images, 10 imagenotions were indirectly used through relations to these imagenotions. 10 imagenotions had only one work step each so that they did not pass the phase one ‘Emergence of ideas’ of our ontology maturing model.
Table 2 shows the aggregated number of work steps of the users for the maturing of imagenotions. All users were able to create relations to other imagenotions. In addition, they added a lot of descriptive information to the created imagenotions. During the workshop, the participants could speak together and discuss available imagenotions. We observed that user 1 (who was very familiar with ontology editing) explained the principes of relations to the other participants. Also, we observed that the usage of links to other web pages in imagenotions improved the background knowledge of the users so that they could in turn add further information, e.g. birthday of persons or relations. Table 3 shows the collaborative usage of imagenotions. Already during the two hours of the evaluation, 24 percent of the imagenotions were used by more than one user and thus entered the phase two of our ontology maturing model “consolidation in communities”. Again, a main reason for that was the possibility of the participants to talk about the created imagenotions.
The participants of the workshop were all experts about the domains of their images. Even in such a small group of six participants, we observed a specialization for different topics of interests. Two participants mainly annotated images showing airplanes and therefore created relevant imagenotions while the other participants mainly created imagenotions for persons and events to annotate the corresponding images. 5
The evalutations showed that our tools and the underlying ontology maturing process model achieved a high level of acceptance by the participants. During the evaluation sessions and in subsequent discussions, we identified missing and requested features for our tools. These features cover better support for consolidation, the distinction of local and global information and a better support for the creation of groups to spezialice for a specific topic of interest, which shall be described in more detail in the following sub sections. 5.1
Based on our ontology maturing process model, the consolidation phase covers combination and refinement of useful ontology elements and the rejection of incorrect or useless ones. Since consolidation is a process of collaborative work, communication between the members of such a community is one of the main functions that help in these processes. In our evaluations, we identified the need for extended communication functionalities, because the participants in the ImageNotion evaluation discussed offline together and in the SOBOLEO evaluation they used the integrated chat functionalites. However, a simple chat is not enough. Based on discussions with the end users, we identied the following four diffent areas in these consolidation processes that require the extension of our application with specific tools: Discussion and Agreement In this area, the participants of the group communicate together discuss about available ontology elements and whether they are useful or not. In addition, in case of similar or even duplicate ontology elements, they discuss whether they should be merged or extended. As the SOBOLEO evaluation showed, a simple chat for all is not enough for that because of too many messages concerning different topics. As a solution, we will extend our tools with a threaded chat system that allows for the separation of discussion topics. In addition, we think that a forum application (e.g. JForum3) is helpful for asynchronous discussions, i.e. when the members of a community are not always online at the same time.
In our evaluations, we had to handle a relatively small number of participants. In
small groups, it is possible to achieve agreements among the members through direct
discussions. In case of bigger groups with ten, fifty or even more participants, direct
aggreements through discussion is no longer possible. As we plan to allow for bigger
groups in our applications, we will extend them with tools that help in voting about
open discussions and in rating the quality of given ideas to achieve agreement.
Execution of Changes This area covers tool support complex operations in the
consolidation phase. Especially for ontology elements with similar meaning (e.g. because they
were created with descriptions in different languages), we see the requirement to
integrate tools that help in handling these complex operations. The merging of two ontology
elements requires updates of all ressources that have been annotated so far with one of
the concerned ontology elements with the newly created one (see e.g. the HCONE
approach ([
Dissemination and Creating Awareness Tools for the dissemination shall help in
informing other members in the group about changes. After the discussion and agreement
about ontology elements and execution of changes, dissemination of these ontology
elements in the community is required to guarantee their usage, e.g. for the annotation
of ressources. Tools like wikis (as proposed by [
Awareness of changes also helps in controlling changes from other users. As
indicated in [
help in notifying them in case of updates. In additions, it could also be helpful to
offer tools that help in undoing changes identifyed as incorrect extensions of ontology
elements – this is one of the instruments of Wikipedia for maturing support [
Both Imagenotion and SOBOLEO currently support a very simple mode of sharing data: all data is shared globally and is jointly edited by everyone. Every statement created is owned by all users and can be seen, edited and deleted by every user. Imagenotion saves who created each statement, but it is not stored when multiple users have the same believe about annotating a resource. This model is similar to that of Wikipedia where a single version of each article is jointly maintained. A competing model is used by social bookmarking sites such as del.icio.us: here each user creates a personal view on the resources. The same tag used for the same resource by multiple persons is stored as two different statements. The personal views of multiple users are connected through the use of common tags.
In the evaluation of SOBOLEO users frequently complained about the lack of such a personal view. One comment representing this line of citique was: “provide a personalized citation browser – only show me the links that I added”.
We are currently working to support a combination of these two edit models for
future versions of the two tools (also taking into account approaches like the HCOME
methodology [
The infrastructure needed to support these use cases differs from well-known access control paradigms (e.g. in file systems) in two main areas: (1) the application of different rights to different parts of rdf-graphs is less well understood than the application of rights to strictly hierarchical data structures (2) the personal view is treated differently than privately editable data. The personal view can be understood as the utterances of a person; hence everyone can only edit her own utterances, but everyone is also free to repeat those of other people or even to make conflicting statemens. This is different from normal access control where private data is simply non-editable for others.
When an ontology is created collaboratively in a larger community, it can be assumed that it will quickly become unwieldy; i.e. that the ontology becomes to large to easily display it in editors, that one user cannot follow all ongoing discussions about changes, that most users are not able or willing to understand the details of parts of the ontology of little concern to them, that there are too many changes happening in quick succession etc. So far we have tried to avoid this problem by intentionally restricting the users to only a small group from a single domain trying to achieve a single joint goal. However, traces of this problem appeared even in our small-scale evaluation when some users started to create sophisticated conceptualizations of the world of military aircraft – to specialized to be of interest to the other users.
As a means to tackle these complexity, there is a strong case to allow for a kind of editable views on the global ontology – smaller ontologies or subgraphs of the ontology that users can commit to. These views could function like thematic user groups on sites like Flickr: e.g. a user interested in military aircraft would join a group specializing in this topic and would then be shown their view, could easily change the relevant concepts and the concepts from this group would be recommended during annotation. For search and browse activities all annotations and ontology would be used by the system, but a preference would be given to those from groups the user is member of. For example when locking at a picture that is annotated with a large number of concepts, a user would see the annotations created by her group(s) and a hint making her aware of other groups that have annotated this particular image. Through these hints she could navigate to the annotations of the other groups. In such a browse scenario, the display of the groups helps in grouping large numbers of annotations and also informs the user about the existence of other groups, thereby fostering consolidation between groups working on related topics.
Introducing such views, however, would come with considerable added complexity, both for the system and the user. At the one hand users would need to understand this added level of abstraction, must be shown and understand how the concepts in the ontology relate to the groups and understand what it means if they leave a group. At the level of the system there is the need not only to manage the groups and their views but also to further support users in finding synergies over groups and to support such complex operations as the merging of ontology created independently. In fact all four consolidation areas identified in section 5.1 apply on groups and views as well. 6
Evaluations of our tools SOBOLEO and ImageNotion have confirmed that our ontology
maturing approach is feasible to enable agile community-driven ontology engineering
for communities of practice. While there are other proposed approaches like [
But a more important result of these evaluation sessions was the guidance for further developments. We are aware that the key for success of ontology maturing support is the right level of complexity: supporting needed actions while retaining ease of use. Therefore, it is crucial that we derive the future development route from actual user needs in a participatory design approach. From the first formative evaluation sessions, we have learnt that we need further developments in the following areas: (1) support for consolidation in all phases (candidate identification, discussion and agreement, execution and dissemination), (2) introduction of an individual scope as the possibility to have diverging private elements, and (3) support for different and diverging microtheories for specific communities/groups.
We will address these issues within our next iteration of development. We also plan to approach the problem of efficient ontology maturing support also in other use cases beyond annotation of resources within the FP7 Integrating Project MATURE4.