In this work, we present an extension of CORE [8], a tool for Collaborative Ontology Reuse and Evaluation. The system receives an informal description of a specific semantic domain and determines which ontologies from a repository are the most appropriate to describe the given domain. For this task, the environment is divided into three modules. The first component receives the problem description as a set of terms, and allows the user to refine and enlarge it using WordNet. The second module applies multiple automatic criteria to evaluate the ontologies of the repository, and determines which ones fit best the problem description. A ranked list of ontologies is returned for each criterion, and the lists are combined by means of rank fusion techniques. Finally, the third component uses manual user evaluations in order to incorporate a human, collaborative assessment of the ontologies. The new version of the system incorporates several novelties, such as its implementation as a web application; the incorporation of a NLP module to manage the problem definitions; modifications on the automatic ontology retrieval strategies; and a collaborative framework to find potential relevant terms according to previous user queries. Finally, we present some early experiments on ontology retrieval and evaluation, showing the benefits of our system.
H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval – information filtering, retrieval models, selection process.
The Web can be considered as a live entity that grows and evolves fast over time. The amount of content stored and shared on the web is increasing quickly and continuously. The global body of multimedia resources on the Internet is undergoing a significant growth, reaching a presence comparable to that of traditional text contents. The consequences of this enlargement result in well known difficulties and problems, such as finding and properly managing all the existing amount of sparse information. To overcome these limitations the so-called “Semantic Web” trend has emerged with the aim of helping machines process information, enabling browsers or other software agents to automatically find, share and combine information in consistent ways. As put by Tim Berners-Lee in 1999, “I have a dream for the Web in which computers become capable of analyzing all the data on the Web – the content, links, and transactions between people and computers. A ‘Semantic Web’, which should make this possible, has yet to emerge, but when it does, the day-to-day mechanisms of trade, bureaucracy and our daily lives will be handled by machines talking to machines. The ‘intelligent agents’ people have touted for ages will finally materialize”.
At the core of these new technologies, ontologies are envisioned as key elements to represent knowledge that can be understood, used and shared among distributed applications and machines. However, ontological knowledge mining and development are difficult and costly tasks that require major engineering efforts. Developing an ontology from scratch requires the expertise of at least two different individuals: an ontology engineer that ensures the correctness during the ontology design and development, and a domain expert, responsible for capturing the semantics of a specific field into the ontology. In this context, ontology reuse becomes an essential need in order to exploit past and current efforts and achievements.
In this scenario, it is also important to emphasize that ontologies,
as well as content, do not stop evolving and growing within the
Web. They are part of its wave of growth and evolution, and they
need to be managed and kept up to date in distributed
environments. In this perspective, the initial efforts to collect
ontologies in libraries [
Novel tools have been recently developed, such as ontology
search engines [
Following this aspiration, the work presented here aims to
enhance ontology retrieval and recommendation, combining
automatic evaluation techniques with explicit users’ opinions and
experiences. This work follows a previous approach for
Collaborative Ontology Reuse and Evaluation over controlled
repositories, named CORE [
Following Leonardo Da Vinci’s words, “Wisdom is the daughter of experience”, our tool aims to take a step forwards for helping users to be wise in exploiting other people’s experience and expertise.
The rest of the paper has been organized as follows. Section 2 summarizes some relevant work related to our system. Its architecture is described in Section 3. Section 4 contains empirical results obtained from early experiments done with a prototype of the system. Finally, several conclusions and future research lines are given in Section 5.
Two well-known scenarios for ontology reuse have been identified in the Semantic Web area. The first one addresses the common problem of finding the most adequate ontologies for a specific domain. The second scenario envisions the not so common but real situation in which Semantic Web applications need to automatically and dynamically find an ontology. In this work, we focus our attention on the fist scenario, where users are the ones who express their information needs. In this scenario, ontology reuse involves several areas such as ontology evaluation, selection, search and ranking.
Several ontology libraries and search engines have been
developed in the last few years to address the problem of ontology
search and retrieval. [
Ontology evaluation can be defined as assessing the quality and the adequacy of an ontology for being used in a specific context, for a specific goal. From our perspective, ontology evaluation constitutes the cornerstone of ontology reuse because it faces the complex task of evaluate, and consequently select the most appropriate ontology on each situation.
An overview of ontology evaluation approaches is presented in
[
When the user and not the application is the one that demands an
ontology, the selection task should be less categorical, returning
not only one but the set of the most suitable results. To sort these
results according to the evaluation criteria, several ontology
ranking measures have been proposed in the literature. Some of
them are presented in [
As it has been shown before, current ontology reuse approaches
take advantage of ontology evaluation, search, retrieval, selection
and ranking methodologies. All these areas provide different
advantages to the process of ontology evaluation and reuse, but
they do not exploit others related to the well known
Recommender Systems [
Collaborative filtering strategies make automatic predictions (filter) about the interests of a user by collecting taste information from many users (collaborating). This approach usually consists of two steps: a) look for users that have a similar rating pattern to that of the active user (the user for whom the prediction is done), and b) use the ratings of users found in the previous step to compute the predictions for the active user. These predictions are specific to the user, differently to those given by more simple approaches that provide average scores for each item of interest, for example based on its number of votes.
Collaborative filtering is a widely explored field. Three main
aspects typically distinguish the different techniques reported in
the literature [
User profile representation and management can be divided into five different tasks: • Profile representation. Accurate profiles are vital for the content-based component (to ensure recommendations are appropriate) and the collaborative component (to ensure that users with similar profiles are in fact similar). The type of profile chosen in this work is the user-item ratings matrix (ontology evaluations based on specific criteria). • Initial profile generation. The user is not usually willing to spend too much time in defining her/his interests to create a personal profile. Moreover, user interests may change dynamically over time. The type of initial profile generation chosen in this work is a manual selection of values for only five specific evaluation criteria. • Profile learning. User profiles can be learned or updated using different sources of information that are potentially representative of user interests. In our work, profile learning techniques are not used. • The source of user input and feedback to infer user interests from information used to update user profiles. It can be obtained in two different ways: using information explicitly provided by the user, and using information implicit observed in the user’s interaction. Our system uses no feedback to update the user profiles. • Profile adaptation. Techniques are needed to adapt the user profiles to new interests and forget old ones as user interests evolve with time. Again, in our approach profile adaptation is done manually (manual update of ontology evaluations). Filtering method. Items or actions are recommended to a user taking into account the available information (item content descriptions and user profiles). There are three main information filtering approaches for making recommendations: • Demographic filtering: Descriptions of people (e.g. age, gender, etc) are used to learn the relationship between a single item and the type of people who like it. • Content-based filtering: The user is recommended items based on the descriptions of items previously evaluated by other users. Content-based filtering is chosen approach in our work (the system recommends ontologies using previous evaluations of those ontologies). • Collaborative filtering: People with similar interests are matched and then recommendations are made.
Matching method. It defines how user interests and item characteristics are compared. Two main approaches can be identified: • User profile matching: people with similar interests are matched before making recommendations. • User profile-item matching: a direct comparison is made between the user profile and the items. The degree of appropriateness of the ontologies is computed by taking into account previous evaluations of those ontologies.
In WebCORE, a new ontology evaluation measure based on collaborative filtering is proposed, considering users’ interests and previous assessments of the ontologies.
As mentioned before, WebCORE is a web application for Collaborative Ontology Reuse and Evaluation. A user logins into the system via a web browser, and, thanks to AJAX technology and the Google Web Toolkit3, dynamically describes a problem domain, searches for ontologies related to this domain, obtains relevant ontologies ranked by several lexical, taxonomic and collaborative criteria, and optionally evaluates by himself those ontologies that he likes or dislikes most.
In this section, we describe the server-side architecture of
WebCORE. Figure 1 shows an overview of the system. We
distinguish three different modules. The first one, the left module,
receives the problem description (Golden Standard) as a full text
or as a set of initial terms. In the first case, the system uses a NLP
module to obtain the most relevant terms of the given text. The
initial set of terms can also be modified and extended by the user
using WordNet [
The first phase of our ontology recommender system is the
Golden Standard definition. As done in the first version of CORE
[
As an improvement, WebCORE includes an internal NLP component that automatically retrieves the most informative terms from a given text. Moreover, we have added a new collaborative component that continuously offers to the user a ranked list with the terms that have been used in those previous problem descriptions in which a given term appears.
In our system, the Golden Standard is described by a set of initial set of terms. These terms can automatically be obtained by the internal Natural Language Processing (NLP) module, which uses a repository of documents related to the specific domain in which the user is interested in. This NLP module accesses to the repository of documents, and returns a list of pairs (lexical entry, part of speech) that roughly represents the domain of the problem. On the other hand, the list of initial (root) terms can be manually specified.
The module also allows the user to expand the root terms using
WordNet [
The final representation of the Golden Standard is defined as a set of terms T (LG, POS, LGP, R, Z) where: • LG is the set of lexical entries defined for the Golden
Standard. • POS corresponds to the different Parts Of Speech considered by WordNet: noun, adjective, verb and adverb. • LGP is the set of lexical entries of the Golden Standard that have been extended. • R is the set of relations between terms of the Golden
has not been obtained by expansion, but is one of the initial terms). • Z is an integer number that represents the depth or distance of a term to the root term from which it has been derived.
T1 = (“genetics”, NOUN, “”, ROOT, 0). T1 is one of the root terms of the Golden Standard. The lexical entry that it represents is “genetics”, its part of speech is “noun”, it has not been expanded from any other term so its lexical parent is the empty string, its relation is “root”, and its depth is 0.
T2 = (“biology”, NOUN, “genetics”, HYPERNYM, 1). T2 is a term expanded from “genetics” (T1). The lexical entry it represents is “biology”, its part of speech is “noun”, the lexical entry of its parent is “genetics”, it has been expanded by the “hypernym“ relation, and the number of relations that separates it from the root term T1 is 1.
Figure 2 shows the interface of the Golden Standard Definition phase. In the left side of the screen, the current list of root terms is shown. The user can manually insert new root terms to this list giving their lexical entries and selecting their parts of speech. The correctness of these new insertions is controlled by verifying that all the considered lexical entries belong to the WordNet repository. Adding new terms, the final Golden Standard definition is immediately updated: the final list of (root and expanded) terms that represent the domain of the problem is shown in the bottom of the figure. The user can also make term expansion using WordNet. He selects one of the terms from the Golden Standard definition and the system shows him all its meanings contained in WordNet (top of the figure). After he has chosen one of them, the system presents him three different lists with the synonyms, hyponyms and hypernyms of the term. The user can then selects one or more elements of these lists and add them to the expanded term list. For each expansion, the depth of the new term is increased by one unit. This will be used later to measure the importance of the term within the Golden Standard: the greater the depth of the derived term with respect to its root term, the less its relevance will be.
In the problem definition phase a collaborative component has been added to the system (right side of Figure 2). This component reads the term currently selected by the user, and searches for all the stored problem definitions that contain it. For each of these problem definitions, the rest of their terms and the number of problems in which they appear are retrieved and shown in the web browser.
With this simple strategy the user is suggested the most popular
terms, fact that could help him to better describe the domain in
which he is interested in. It is very often the case that a person has
very specific goals or interests, but does not know how to
correctly explain/describe them, and how to effectively find
solutions for them. With the retrieved terms, the user might
discover new ways to describe the problem domain and obtain
better solutions in the ontology recommendation phase.
This follows somehow the ideas of the well known folksonomies4.
The term “folksonomy” is a combination of “folk” and
“taxonomy”, and was firstly used by Thomas Vander Wal [
and Communication through Shared Metadata. http://www.adammathes.com/academic/computer-mediatedcommunication/folksonomies.html a discussion on a mailing list about the system of organization developed in Delicious5 and Flickr6. It is associated to those information retrieval methodologies consisting of collaboratively generated, open-ended labels that categorize content.
Although they suffer from problems of imprecision and ambiguity, techniques employing free-form tagging encourage users to organize information in their own ways and actively interact with the system.
Once the user has selected the most appropriate set of terms to
describe the problem domain, the tool performs the processes of
ontology retrieval and ranking. These processes play a key role
within the system, since they provide the first level of information
to the user. To enhance the previous approaches of CORE, an
adaptation of traditional Information Retrieval techniques have
been integrated into the system. Our novel strategy to ontology
retrieval can be seen as an evolution of classic keyword-based
retrieval techniques [
The queries supported by our model are expressed using the terms
selected during the Golden Standard definition phase.
In classic keyword-based vector-space models for information
retrieval [
Analogously, in our model, the terms included in the Golden Standard can be weighted to indicate the relative interest of the 5 del.icio.us - social bookmarking, http://del.icio.us/ 6 Flickr - photo sharing, http://www.flickr.com/ user for each of the terms to be explicitly mentioned in the ontologies. In our system, these weights are automatically assigned considering the depth measure of each of the terms included in the Golden Standard.
Let T be the set of all terms defined in the Golden Standard
definition phase. Let di be the depth measure associate with each
term ti ∈ T. Let q be query vector extracted from the Golden
Standard definition, and let wi be the weight associated to each of
these terms, where for each ti ∈ T, wi ∈ [
To carry out the process of ontology retrieval, the approach is focused on the lexical level, retrieving those ontologies that contain a subset of the terms expressed by the user during the Golden Standard definition. To compute the matching, two different options are available within the tool: search for exact matches and search for matches based on the Levenshtein distance between two terms.
In both cases, the query execution returns a set of ontologies that satisfy user requirements. Considering that not all the retrieved ontologies fulfil the same level of satisfaction, it is the system task to sort them and present the ranked list to the user.
Once the list of ontologies is formed, the ontology-search engine computes a semantic similarity value between the query and each ontology as follows. We represent each ontology in the search space as an ontology vector oj ∈ O, where oji is the mean of the term ti similarities with all the matched entities in the ontology if any matching exists, and zero otherwise.
The components oji are calculated as: ∑ w(m ji )
M ji o ji = M ji ∑ w(mi )
Mi where Mji is the set of matches of the term ti in the ontology oj, w(mji) represents the similarities between the term ti and the entities of the ontology oj that matches with it, Mi is the set of matches of the term ti within all the ontologies and w(mi) represents the weights of each of these matches.
For example, if we define in the Golden Standard a term “acid”, this term may return several matches in the same ontology with different entities as: “acid”, “amino acid”, etc. In order to establish the appropriate weight in the ontology vector, oij, the goal is to compute the number of matches of one term in the whole repository of ontologies and give more relevance to those ontologies that have matched that specific term more times. Due to the way in which the vector oj is constructed, each component oij contains specific information about the similarity between the ontology and the corresponding term ti. To compute the final similarity between the query vector q and the ontology vector oj, the vectorial model calculates the cosine measure between both vectors. However, if we follow the traditional vectorial model, we will only be considering the difference between the query and the ontology vectors according to the angle they form, but not taking into account their dimensions. Thus, to overcome this limitation, the above cosine measure used in the vectorial model has been replaced by the simple dot product. Hence, the similarity measure between an ontology oj and the query q is simply compute as follows:
sim(q, o j ) =q ⋅ o j
If the knowledge in the ontology is incomplete, the ontology
ranking algorithm performs very poorly. Queries will return less
results than expected, the relevant ontologies will not be retrieved,
or will get a much lower similarity value than it should. For
instance, if there are ontologies about “restaurants”, and “dishes”
are expressed as instances in the corresponding Knowledge Base
(KB), a user searching for ontologies in this domain may be also
interested in the instances and literals contained in the KB. To
cope with this issue, our ranking model combines the similarity
obtained from the terms that belong to the ontology with the
similarity obtained from the terms that belong to the KB using the
adaptation of the vector space model explained before.
On the other hand, the combination of outputs of several search
engines has been a widely addressed research topic in the
Information Retrieval field [
si = v5i Following this idea, the final score is computed as: sO × sim(q, o) + skb × sim(q, kb) For future work, we are considering to set si using statistical information about the knowledge contained in the ontologies, the knowledge contained in the KBs and the information requested by the user during the Golden Standard definition phase. Figure 3 shows the system recommendation interface. At the left side the user can select the matching methodology (fuzzy or exact), the search spaces (ontology entities and knowledge base entities), and the weight or importance given to each of the previously selected search spaces. In the right part the user can visualize the ontology and navigate across it. Finally, the middle of the interface presents the list of ontologies selected for the user to be evaluated during the collaborative evaluation phase.
The third and last phase of the system is compound of a novel
ontology recommendation algorithm that exploits the advantages
of Collaborative Filtering [
In WebCORE, user evaluations are represented as a set of five
different criteria [
• Correctness: specifies whether the information stored in the ontology is true, independently of the domain of interest. • Readability: indicates the non-ambiguous interpretation of the meaning of the concept names. • Flexibility: points out the adaptability or capability of the ontology to change. • Level of formality: highly informal, semi-informal, semiformal, rigorously-formal. • Type of model: upper-level (for ontologies describing general, domain-independent concepts), core-ontologies (for ontologies that contain the most important concepts on a specific domain), domain-ontologies (for ontologies that broadly describe a domain), task-ontologies (for ontologies focused on generic types of tasks or activities) and application-ontologies (for ontologies describing a domain in an application-dependent manner).
The above criteria can have discrete numeric or non-numeric values. The user’s interests are expressed like a subset of these criteria, and their respective values, meaning thresholds or restrictions to be satisfied by user evaluations. Thus, a numeric criterion will be satisfied if an evaluation value is equal or greater than that expressed by its interest threshold, while a non-numeric criterion will be satisfied only when the evaluation is exactly the given threshold (i.e. in a Boolean or yes/no manner).
According to both types of user evaluation and interest criteria, numeric and Boolean, the recommendation algorithm will measure the degree in which each user restriction is satisfied by the evaluations, and will recommend a ranked ontology list according to similarity measures between the thresholds and the collaborative evaluations. To create the final ranked ontology list the recommender module follows two phases. In the first one it calculates the similarity degrees between all the user evaluations and the specified user interest criteria thresholds. In the second one it combines the similarity measures of the evaluations, generating the overall rankings of the ontologies.
Figure 4 shows all the previous definitions and ideas, locating them in the graphical interface of the system. On the left side of the screen, the user introduces the thresholds for the recommendations and obtains the final collaborative ontology ranking. On the right side, the user adds new evaluations for the ontologies and checks evaluations given by the rest of the users.
As mentioned before, a user evaluates an ontology considering
five different criteria that can be divided in two different groups:
a) numeric criteria (‘correctness’, ‘readability’ and ‘flexibility’),
which take discrete numeric values [
E1 2 0 E1 3 0 5
E2 0 0 E2 4 1 5
Evaluations Evaluations E3 2 2 E3 5 4 5
E4 0 0 E4 5 5 5
E5 2 0 E5 2 0 4
E6 0 0 E6 0 0 0 represented by specific non-numeric values that can be or not satisfied by the ontology.
Taking into account the previous definitions, user interests will be a subset of the above criteria and their respective values representing the set of thresholds that should be reached by the ontologies. Given a set of user interests, the system will size up all the stored evaluations, and will calculate their similarity measures. To explain these similarities we shall use a simple example of six different evaluations (E1, E2, E3, E4, E5 and E6) of a given ontology. In the explanation we shall distinguish between the numeric and the Boolean criteria. We start with the Boolean ones, assuming two different criteria, C1 and C2, with three possible values: “A”, “B” and “C”. In Table 1 we show the threshold values established by a user for these two criteria, “A” for C1 and “B” for C2, and the six evaluations stored in the system. In this case, because of the threshold of a criterion n is satisfied or not by a certain evaluation m, their corresponding similarity measure is simply 0 if they have the same value, and 2 otherwise.
⎧0 similaritybool (criterionmn ) = ⎨ ⎩2 if evaluationmn ≠ thresholdmn if evaluationmn = thresholdmn The similarity results for the Boolean criteria of the example are shown in Table 3.
= 1 + similarityn*um (criterionmn )· penaltynum (thresholdmn ) ∈ [
The similarity assessment is based on the distance between the value of the criterion n in the evaluation m, and the threshold indicated in the user’s interests for that criterion. The more the value of the criterion n in evaluation m overcomes the threshold, the greater the similarity value shall be.
Specifically, following the expression below, if the difference dif = (evaluation – threshold) is equal or greater than 0, we assign a positive similarity in (0,1] that depends on the maximum difference maxDif = (maxValue – threshold) we can achieve with the given threshold; and else, if the difference dif is lower than 0, we give a negative similarity in [-1,0), punishing the distance of the value with the threshold.
⎧ 1 + dif ⎪ ⎪1 + maxDif similarityn*um (criterionmn ) = ⎨ ⎪ dif
E1 1/4 1/6 1
E2 2/4 2/6 1
Evaluations E3 3/4 5/6 1
E4 3/4 1 1
E5 -1/3 1/6 -1/5
E6 -1 1/6 -1 Comparing the evaluation values of Table 4 with the similarity values of Table 5, the reader may notice several important facts: 1. Evaluation E4 satisfies criteria C4 and C5 with evaluations of 5.
Applying the above expression, these criteria receive the same similarity of 1. However, criterion C4 has a threshold of 0, and C5 has a threshold equal to 5. As it is more difficult to satisfy the restriction imposed to C5, this one should have a greater influence in the final ranking. 2. Evaluation E6 gives an evaluation of 0 to criteria C3 and C5, not satisfying either of them and generating the same similarity value of -1. Again, because of their different thresholds, we should distinguish their corresponding relevance degrees in the rankings.
For these reasons, a threshold penalty is applied, reflecting how difficult it is to overcome the given thresholds. The more difficult to surpass a threshold, the lower the penalty value shall be. penaltynum (threshold ) =
∈ (0,1] Table 6 shows the threshold penalty values for the three numeric criteria and the six evaluations of the example. For the numeric criteria, the evaluations can overcome the thresholds to different degrees. Table 4 shows the thresholds established for criteria C3, C4 and C5, and their six available evaluations. Note that E1, E2, E3 and E4 satisfy all the criteria, while E5 and E6 do not reach some of the corresponding thresholds. In this case, the similarity measure has to take into account two different issues: the degree of satisfaction of the threshold, and the difficulty of achieving its value. Thus, the similarity between the value of criterion n in the evaluation m, and the threshold of interest is divided into two factors: 1) a similarity factor that considers whether the threshold is surpassed or not, and, 2) a penalty factor which penalizes those thresholds that are easier to be satisfied.
E1 4/6 1/6 1 E1 As a preliminary approach, we calculate the similarity between an ontology evaluation and the user’s requirements as the average of its N criteria similarities.
similarity(evaluationm ) = 1 N
∑ similarity(criterionmn )
N n=1 A weighted average could be even more appropriate, and might make the collaborative recommender module more sophisticated and adjustable to user needs. This will be considered for a possible enhancement of the system in the continuation of our research.
Once the similarities are calculated taking into account the user’s interests and the evaluations stored in the system, a ranking is assigned to the ontologies.
The ranking of a specific ontology is measured as the average of its M evaluation similarities. Again, we do not consider different priorities in the evaluations of several users. We have planned to include in the system personalized user appreciations about the opinions of the rest of the users. Thus, for a certain user some evaluations will have more relevance than others, according to the users that made it.
ranking (ontology) = = 1 M
∑ similarity(evaluationm ) M m=1 1 M N
∑ ∑ similarity(criterionmn )
MN m=1 n=1 Finally, in case of ties, the collaborative ranking mechanism sorts the ontologies taking into account not only the average similarity between the ontologies and the evaluations stored in the system, but also the total number of evaluations of each ontology, providing thus more relevance to those ontologies that have been rated more times.
M
ranking (ontology)
In this section, we present some early experiments that attempt to measure: a) the gain of efficiency and effectiveness, and the b) increment of users’ satisfaction obtained with the use of our system when searching ontologies within a specific domain. The scenario of the experiments was the following. A repository of thirty ontologies was considered and eighteen subjects participated in the evaluations. They were Computer Science Ph.D. students of our department, all of them with some expertise in modeling and exploitation of ontologies. They were asked to search and evaluate ontologies with WebCORE in three different tasks. For each task and each student, one of the following problem domains was selected: • Family. Search for ontologies including family members: mother, father, daughter, son, etc. • Genetics. Search for ontologies containing specific vocabulary of Genetics: genes, proteins, amino acids, etc. • Restaurant. Search for ontologies with vocabulary related to restaurants: food, drinks, waiters, etc.
In the repository, there were six different ontologies related to each of the above domains, and twelve ontologies describing other no related knowledge areas. No information about the domains and the existent ontologies was given to the students.
Tasks 1 and 2 were performed first without the help of the collaborative modules of the system, i.e., the term recommender of the problem definition phase and the collaborative ranking of the user evaluation phase. After all users finished the previous ontology searches and evaluations, task 3 was done with the collaborative components activated. For each task and each student, we measured the time expended, and the number of ontologies retrieved and selected (‘reused’). We also asked the users about their satisfaction (in a 1-5 rating scale) about each of the selected ontologies and the collaborative modules. Tables 8 and 9 contain a summary of the obtained results. Note that measures of task 1 are not shown. We have decided not to consider them for evaluation purposes because we discern the first task as a learning process of the use of the tool, and its time executions and number of selected ontologies as skewed no objective measures.
To evaluate the enhancements in terms of efficiency and effectiveness, we present in Table 8 the average number of reused ontologies and the average execution times for task 2 and 3. The results show a significant improvement when the collaborative modules of the system were activated. In all the cases, the students made use of the terms and evaluations suggested by others, accelerating the processes of problem definition and relevant ontology retrieval. On the other hand, table 9 shows the average degrees of satisfaction revealed by the users about the retrieved ontologies and the collaborative modules. Again, the results evidence positive applications of our approach.
Task
3
% improvement
Initial term recommendation
Final ontology
ranking
In this paper, a web application for ontology evaluation and reuse has been presented. The novel aspects of our proposal include the use of WordNet to help users to define the Golden Standard; a new ontology retrieval technique based on traditional Information Retrieval models; rank fusion techniques to combine different ontology evaluation measures; and two collaborative modules: one that suggests the most popular terms for a given domain, and one that recommends lists of ontologies with a multi-criteria strategy that takes into account user opinions about ontology features that can only be assessed by humans.