<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Validating Top-level and Domain Ontology Alignments using WordNet</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Daniela Schmidt</string-name>
          <email>daniela.schmidt@acad.pucrs.br</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Cassia Trojahn</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Renata Vieira</string-name>
          <email>renata.vieira@pucrs.br</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Mouna Kamel</string-name>
          <email>mouna.kamelg@irit.fr</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Institut de Recherche en Informatique de Toulouse</institution>
          ,
          <country country="FR">France</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Pontifical Catholic University of Rio Grande do Sul</institution>
          ,
          <country country="BR">Brazil</country>
        </aff>
      </contrib-group>
      <fpage>119</fpage>
      <lpage>130</lpage>
      <abstract>
        <p>Matching domain and top-level ontologies is an important task but still an open problem in the ontology matching field, particularly due to their different levels of abstraction. Beyond that, validating candidate alignments is crucial before exploiting them within ontology construction and integration processes involving such kinds of ontologies. This paper concerns the automatic validation of candidate alignments between top and domain ontologies, exploiting WordNet background knowledge. We apply our approach for validating alignments generated by a set of matching tools and discuss the results against manual validation. Our experiments are based on one domain ontology and three well-known top-level ontologies.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        According to their “level of generality”, ontologies can be classified in different
categories [
        <xref ref-type="bibr" rid="ref8">Guarino 1998</xref>
        ]. Top-level ontologies describe very general concepts (e.g., space,
time, object, etc.), which are independent of a particular domain, and are usually equipped
with a rich axiomatic layer; domain ontologies describe the entities related to a domain
(e.g., biology or aeronautic). While the rich semantics and formalization of top-level
ontologies are important requirements for ontology building [
        <xref ref-type="bibr" rid="ref20">Mika et al. 2004</xref>
        ], they act as
well as semantic bridges supporting very broad semantic interoperability between domain
ontologies [
        <xref ref-type="bibr" rid="ref18">Mascardi et al. 2010</xref>
        ]. For grounding domain concepts, one is required to find
relationships with concepts from a top-level ontology, i.e., to match them.
      </p>
      <p>
        There have been many proposals to the matching task in the literature
[
        <xref ref-type="bibr" rid="ref6">Euzenat and Shvaiko 2007</xref>
        ], which exploit in different ways the knowledge encoded
within each ontology when identifying correspondences between their features or
structures. However, most efforts have been particularly dedicated to domain ontologies and
the problem of matching domain and top-level ontologies has been addressed to a lesser
extent. Matching domain and top-level ontologies poses different challenges, in
particular due to their different levels of abstraction. This is a complex task, even manually, that
requires to deeply identify the semantic context of concepts. It involves going beyond the
frontiers of the knowledge encoded in the ontologies and, in particular, the identification
of subsumption relations. The latter is in fact largely neglect by most matchers. Beyond
that, the validation of alignments is a crucial step before exploiting them within ontology
building and integration processes. While manual validation is a time-consuming task
requiring expert judges, automatic ones generally rely on gold standards, which are not
always available or may suffer of imperfections or low coverage. Exploiting background
knowledge can be an alternative. However, knowledge on top level ontologies is highly
specialized and requires resources covering different levels of abstraction.
      </p>
      <p>
        This paper concerns the automatic validation of candidate alignments between
top-level and domain ontologies, relying on background knowledge from WordNet
[
        <xref ref-type="bibr" rid="ref21">Miller 1995</xref>
        ] and on the notion of context, which has been exploited in different ways
in ontology matching [
        <xref ref-type="bibr" rid="ref16">Maedche and Staab 2002</xref>
        ,
        <xref ref-type="bibr" rid="ref28">Wang 2011</xref>
        ,
        <xref ref-type="bibr" rid="ref25">Schadd and Roos 2012</xref>
        ,
        <xref ref-type="bibr" rid="ref5">Djeddi and Khadir 2014</xref>
        ]. Contexts are constructed from all available information about
an ontology entity, including entity naming (ID), annotation properties (labels and
comments) and information on the neighbors of entities (super and sub-concepts). They are
used for disambiguating the senses that better express the meaning of ontology entities,
before looking for relations between these senses in WordNet in order to validate or not a
correspondence between those entities. Although our approach can be applied for
validating any kind of alignment (i.e., domain ontology alignments), it is particularly suitable for
validating alignments which are rich in subsumption relations, what is typically the case
when matching domain and top-level ontologies, since it takes advantage of the network
of conceptual-semantic and lexical relations in WordNet (in particular, hypernym
relations). Furthermore, although this approach can be seen as a matching approach itself,
our motivation here is to automatically validate alignments in the lack of gold standards.
We apply our approach for validating alignments generated by a set of matching tools
        <xref ref-type="bibr" rid="ref26">(see more details in [Schmidt et al. 2016])</xref>
        and discuss the results against a manual
validation. Our experiments are based on three well-known top-level ontologies (DOLCE,
GFO, and SUMO) and one domain ontology from one of the most used data set in the
Ontology Alignment Evaluation Initiative (OAEI) campaigns (the Conference data set).
The rest of the paper is organised as follows. x2 introduces WordNet structure, top-level
ontologies and ontology matching. x3 describes the validation approach. x4 presents the
experiments. x5 discusses related work and, finally, x6 concludes the paper.
      </p>
    </sec>
    <sec id="sec-2">
      <title>2. Background</title>
    </sec>
    <sec id="sec-3">
      <title>2.1. WordNet</title>
      <p>
        WordNet [
        <xref ref-type="bibr" rid="ref21">Miller 1995</xref>
        ] is a general-purpose large lexical database of English. Nouns,
verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each
expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic
and lexical relations. A synset denotes a concept or a sense of a group of terms.
WordNet also provides textual descriptions of the concepts (gloss) containing definitions and
examples. For instance, for the concept “Poster”, one of the associated WordNet synsets
(SID-06793426-N) groups the synonyms “poster, posting, placard, notice, bill, card”,
together with a gloss “a sign posted in a public place as an advertisement; a poster advertised
the coming attractions”.
      </p>
    </sec>
    <sec id="sec-4">
      <title>2.2. Top-level ontologies</title>
      <p>
        A top-level ontology is a high-level and domain independent ontology. The concepts
expressed are intended to be basic and universal to ensure generality and expressivity for a
wide range of domains. It is often characterized as representing common sense concepts
and concerns concepts which are meta, generic, abstract and philosophical. There are two
approaches for the use of top-level ontologies. The top-down approach uses the ontology
as a foundation for deriving concepts in the domain ontology, taking advantage of the
knowledge and experience already expressed in the top-level ontology. In a bottom-up
approach, one usually matches a new or existing domain ontology to a top-level
ontology. This paper concerns bottom-up approaches. Several top-level ontologies have been
proposed in the literature. The reader can refer to [
        <xref ref-type="bibr" rid="ref17">Mascardi et al. 2007</xref>
        ] for a review of
them. Here, we briefly introduce some well-known and largely used top-level ontologies
which are used further in our experiments:
      </p>
      <p>
        DOLCE [
        <xref ref-type="bibr" rid="ref7">Gangemi et al. 2002</xref>
        ]: Descriptive Ontology for Linguistic and Cognitive
Engineering has been proposed at LOA (Laboratory for Applied Ontology). Its focus is to
grasp the underlying categories of human cognitive tasks and the socio-cultural
environment. It is an ontology of particulars and includes concepts such as abstract quality,
abstract region, physical object, process, and so on. Here, we consider the version
DOLCE Lite1, which is composed by 37 concepts and 70 object properties.
GFO [
        <xref ref-type="bibr" rid="ref9">Herre et al. 2007</xref>
        ]: General Formal Ontology is a top-level ontology for
conceptual modeling that has been proposed by the Onto-Med Research Group. It includes
elaborations of categories such as objects, processes, time and space, properties,
relations, roles, functions, facts, and situations. The version we consider here, GFO Basic2,
is composed by 45 Concepts and 41 Object properties.
      </p>
      <p>
        SUMO [
        <xref ref-type="bibr" rid="ref24">Niles and Pease 2001</xref>
        ]: Suggested Upper Merged Ontology has been proposed
by The IEEE Standard Upper Ontology Working Group. It provides definitions for
general-purpose terms and acts as a foundation for more specific domain ontologies and
is used for research and applications in search, linguistics and reasoning. We consider
the version SUMO3 which is composed by 4.558 concepts and 778 object properties.
      </p>
    </sec>
    <sec id="sec-5">
      <title>2.3. Ontology matching</title>
      <p>
        The process of finding correspondences between ontology entities is known as
ontology matching. It takes as input two ontologies os (source) and ot (target) and an
(possibly empty) alignment A to be completed, and determines as output an alignment A0,
i.e., a set of correspondences. Here, we borrow the definition of correspondence from
[
        <xref ref-type="bibr" rid="ref6">Euzenat and Shvaiko 2007</xref>
        ]:
Definition 1 (Correspondence) A correspondence can be defined as &lt;es,et ,r,n&gt;, such
that: es and et are entities (e.g., concepts, properties, instances) of os and ot , respectively;
r is a relation holding between two entities es and et , (for instance, equivalence,
subsumption, disjointness, overlapping); and n is a confidence measure number in the [0;1] range.
The confidence assigns a degree of trust on the correspondence from the matcher.
      </p>
      <p>
        Different matching approaches have emerged from the literature
[
        <xref ref-type="bibr" rid="ref13">Kalfoglou and Schorlemmer 2003</xref>
        ,
        <xref ref-type="bibr" rid="ref6">Euzenat and Shvaiko 2007</xref>
        ]. While
terminological methods lexically compare strings (tokens or n-grams) used in naming entities (or in
the labels and comments of entities), semantic methods utilize model-theoretic semantics
to determine whether or not a correspondence exists between 2 entities. Approaches may
consider the internal ontological structure, such as the range of their properties, their
cardinality, and their transitivity and/or symmetry features, or alternatively the external
1http://www.loa.istc.cnr.it/old/DOLCE.html
2http://onto.eva.mpg.de/gfo-bio/gfo-bio.owl
3http://www.adampease.org/OP/SUMO.owl
ontological structure, such as the position of the 2 entities within the ontological
hierarchy. The instances of concepts can also be compared using statistical, probabilistic and
linguistic approaches. Moreover, ontology matching systems rely on several approaches.
      </p>
    </sec>
    <sec id="sec-6">
      <title>3. Validation approach</title>
      <p>Our validation approach is based on the intuition that a correspondence c from an
alignment A, expressing a relation r (in particular, equivalence or sumbumption) between the
entities ed and et of a domain ontology od and a top-level ontology ot , respectively, may
exist as a semantic relation in WordNet (synonymy or hypernymy). Given ed and et
concepts in the ontologies, we have first to retrieve the most appropriated WordNet synsets
and then check if there exists a relation r between such synsets. Our approach thus is
based on two main steps : (i) synset disambiguation and (ii) synset relation identification.
Synset disambiguation. In order to select the synset that better expresses the meaning
behind the ontology concept, we adopt the notion of context. A context is constructed
from all information available about an ontology entity, including entity naming (ID),
annotation properties (usually labels and comments) and information on the neighbors
(super and sub-concepts). Given Supe and Sube, the sets of terms denoting the
superconcepts and sub-concepts of the entity e, and Anne the set of terms from its annotations,
a naive strategy for building a context (ctx) considers these sets as a bag of words. In a
preprocessing phase, the annotations are tokenized and stop words and special characters are
removed. Hence, the context ctxe of an entity e is a bag of words from the pre-processed
sets Supe, Sube and Anne, including e itself (i.e., its ID) :</p>
      <p>ctxe = fe; w j w 2 Supe [ w 2 Sube [ w 2 Anneg
We take ctxe for retrieving the closer WordNet synset with respect to the meaning of
e. For that, we overlap ctxe with the context of each retrieved synsetie when looking for
e in WordNet, and select the set with higher intersection. We compute the number of
words occurring in both sets, based on the exact match between words. Given Termsie
the set of terms in a synsetie and Glossie the corresponding set of terms from the gloss
(i.e, textual description containing definitions and examples) associated to the synset, the
context ctxsynsetie of the synsetie is defined as follows:</p>
      <p>ctxsynsetie = fe; w j w 2 Termsie [ w 2 Glossieg</p>
      <p>The overlap between ctxe and ctxsynsetie is calculated as follows (where n is the
number of synsets of e):
overlap(ctxe; ctxsynsetie ) = max jctxe \ ctxsynsetie j
1 i n
Synset relation identification. Having selected synsetied and synsetiet , the closest
WordNet synsets expressing, respectively, the meaning of the entities ed and et , we verify if
synsetied = synsetiet . In this case, we assume that ed and et are equivalent (r = equivalence).
Otherwise, we construct the set Super synsetied which is composed of all the hypernym
synsets of synsetied of rank k (k being the maximum length of the path from synsetied to
one of its hypernym synsets in WordNet, based on a depth-first search strategy). The
intuition here is that if the concept in the top-level ontology belongs to the set of hypernyms
of the domain concept, then a subsumption relation exists between them. Hence, synsetied
subsumes synsetiet if synsetiet belongs to the set Super synsetied (r = subsumption).
Example. Given &lt;PresentationCon f erence,EventGFO &gt;, one correspondence to be
validated (taken from the ontologies used in our experiments) and the sets:</p>
      <p>SupPresentation = fconference, document, contribution, invited, talkg (from the original
set fConference document, Conference contribution, Invited talkg);
SubPresentation = 0/ (the domain concept has no sub-concepts);
AnnPresentation = 0/ (the domain concept has no annotations);
SupEvent = findividual, concrete, processual, structure, occurrentg (from the original
set findividual, concrete, processual structure, occurrentg);
SubEvent = 0/ (the domain concept has no sub-concepts);
AnnEvent = fevent, processual, structure, processg (from the annotation “Events are
processual structures comprising a process ...”);
The following contexts can be constructed:
ctxpresentation = fpresentation, conference, document, contribution, invited, talkg
ctxevent = fevent, individual, concrete, processual, structure, occurrent, processg</p>
      <p>Overlapping the context of the concepts with all their WordNet synsets, the closer
synsets are = fSID-01048697-N (presentation)g and fSID-00029378-N (event)g. As they
are different synsets, the equivalence between them can not be verified. Hence, we look
for the Super synset of the concept presentation (from domain ontology) which is =
fSID01048697-N (presentation), SID-01027379-N (ceremony), SID-00407535-N (activity),
SID-00030358-N (act, deed, human action, human activity), SID-00029378-N (event)g.
As there is an overlap between this Super synset and the synset for event
(SID-00029378N (event)) then we assume that PresentationCon f erence subsumes EventGFO.</p>
    </sec>
    <sec id="sec-7">
      <title>4. Experiments</title>
    </sec>
    <sec id="sec-8">
      <title>4.1. Data set and matchers</title>
      <p>
        OAEI Conference data set. The OAEI Conference set4 contains 16 ontologies
covering the domain of conference organization. We have chosen this data set because
it contains expressive ontologies and is one of the most popular data set in the field
[
        <xref ref-type="bibr" rid="ref3">Cheatham and Hitzler 2014</xref>
        ]. In our experiments, we have used only the Conference
ontology5. This ontology has 60 concepts, 46 object properties and 18 data properties.
Ontology matching tools. A set of tools, publicly available, from previous OAEI
campaigns (not limited to Conference track top participants), and implementing different
matching strategies was selected. Even though they are not exhaustive and were not
exactly developed for that purpose, their output might help us to investigate the problem
of aligning domain and top-level ontologies. Aroma6 is a hybrid tool based on association
rules; Falcon-AO7 applies linguistic and structural approaches, as Lily8, which includes
debugging strategies; LogMap9 applies logical reasoning and repair strategies and its
variant LogMap-Lite is essentially based on string similarities; MaasMatch adopts a
similar4http://oaei.ontologymatching.org
5http://oaei.ontologymatching.org/2015/conference/data/Conference.owl
6https://exmo.inrialpes.fr/software/aroma/
7http://ws.nju.edu.cn/falcon-ao/
8http://cse.seu.edu.cn/people/pwang/lily.htm
9https://www.cs.ox.ac.uk/isg/tools/LogMap/
ity cube and a disambiguation phase as described in [
        <xref ref-type="bibr" rid="ref25">Schadd and Roos 2012</xref>
        ];
WeSeEMatch10 uses web search results for improving similarity measures; WikiMatch11 uses
Wikipedia as external knowledge source and YAM++12 applies linguistic, graph-based
and machine learning approaches. MaasMatch and YAM++ use WordNet as background
knowledge (see x5). All the tools were run with their default configuration settings.
      </p>
    </sec>
    <sec id="sec-9">
      <title>4.2. Results and discussion</title>
      <p>
        Manual analysis. We first qualitatively analysed how the available matching tools
perform when aligning domain with top-level ontologies, based on a manual validation of
the generated alignments, by 3 evaluators. For our experiments, we ran each of the above
mentioned systems for the pairs composed by the Conference ontology against each
toplevel ontology. We then merge the alignments generated by the matchers, resulting in 28
correspondences, and submitted the resulting merge to the analysis of three evaluators.
The evaluators are researchers that have common-sense knowledge about the domain of
conferences, with a strong background in Computer Science and well-familiarised with
ontology matching. The evaluators analysed each correspondence and selected one type
of relation (Equivalent, Subsumption or None), according to the relation they judged as
correct, based on the concept description provided in the top-level ontology
        <xref ref-type="bibr" rid="ref26">(see more
details in [Schmidt et al. 2016])</xref>
        . Regarding the evaluators judgement, there was a total
agreement among them in 21 (out of 28) correspondences. For 15 of them, no relation
has been identified so that more than a half of the automatically aligned concepts were
considered neither equivalent or subsumed. In 3 cases there was total agreement
regarding subsumption, and in 3 cases total agreement for equivalence. For the 7 pairs, a partial
agreement was achieved as subsumption (see Table 2, column ‘Manual evaluation’). In
the following, we quantitatively analyse the results obtained by the matchers, using
precision and considering 4 different sets:
      </p>
      <p>P1 considers the cases of total agreement among the judges (21 correspondences),
where the correspondence has been marked either as equivalent or subsumed;
P2 considers both total and partial agreements (for equivalent or subsumed);
P3 considers total agreement for equivalences (matchers generated only equivalences);
P4 considers both total and partial agreements only for equivalences.</p>
      <p>Table 1 shows the matchers results for these 4 sets, in terms of precision. From
9 matchers, 2 of them (MaasMatch and YAM++) have used WordNet in their matching
approaches (in particular, MassMatch with a disambiguation phase). As expected, the
best results were obtained for P2 for both systems. However, the best results for this
set have been obtained by LogMap. Although, ideally, avoiding a bias in the evaluation
is highly desirable and the background resources used in the validation process should
not be used for validating correspondences generated on basis of them, our evaluation is
not limited to matchers relying on WordNet. We have used a set of available matchers
using different matching strategies and tried to maximize the testing set by merging their
output. We can in fact observe that MassMatch and YAM++ were able to generate more
correspondences than the other systems (with LogMap and its variant coming just behind).
Contrary to what would be expected, these systems (and all others, in fact) were not able
to generate subsumption (even though some have been designed to). They generated only
equivalences, even when they were in fact subsumptions. This is actually an important
distinction that our validation tries to identify.</p>
      <p>Automatic validation. We have submitted the 28 automatically generated
correspondences to the automatic validation and compared the results with the manual evaluation
(Table 2). Although, by definition, top-level ontologies should be able to cover most
domain concepts, the matchers were not able to generate correspondences for most of the
concepts (42 in a total of 60). A total of 8 pairs were automatically validated. From
these pairs, 5 were correspondences validated manually by the evaluators. While the
automatic validation distinguishes relations of equivalence and subsumption, it is still not
able to distinguish as much as the manual analysis. The comparison of the automatic
with the manual validation is given in terms of precision, recall, f-measure, and accuracy,
in Table 3. Since the matchers do not distinguish equivalences and subsumptions, when
evaluating our automatic approach, P3 and P4 have not been considered.</p>
      <p>Looking at the results in detail (Table 2), we see 5 pairs with agreement between
automatic and manual validation:</p>
      <p>Equivalence: Poster and Poster (SUMO), Publisher and Publisher (SUMO);
Subsumption: Organizer and Organism (SUMO), Presentation and Event (GFO), and
Person and Individual (GFO).</p>
      <p>For the first 4 pairs above, the evaluators totally agreed on equivalence or
subsumption relations and for the fifth pair, they partially agreed on subsumption. One of
them has been generated by MassMatch (pair Presentation and Event(GFO)) and the other
4 by YAM++. In the automatically validated relations, the pairs involving an equivalence
(Poster and Publisher), the same WordNet synset for both top-level and domain concepts
was selected in the disambiguation step – respectively, SID-06793426-N (poster, posting,
placard, notice, bill, card) and SID-08062623-N (publisher, publishing house,
publishing firm, publishing company). For subsumption relations, for the pair Organizer and
Organism(SUMO), the selected synsets were not the same – respectively,
SID-10383237N (organizer, organiser, arranger) and SID-00004475-N (organism, being) but the domain
synset is related to the top-level synset via a hypernym relation (Organizer is-a Organism).
The same was observed for the pairs Presentation and Event, and Person and Individual.</p>
      <p>For the 3 pairs of concepts automatically validated but manually annotated as
‘None’, the concept Abstract appears in 2 pairs, and the third pair is related to the concept
Workshop. For the first pair involving Abstract (GFO), 2 contexts intersect both domain
and top-level concept contexts with the same number of words. However, the wrong
synset was selected for the domain concept. While Abstract in the domain ontology
corresponds to the sense SID-06468951-N (outline, synopsis, abstract, precis) and Abstract
in the top-level ontology to the sense SID-05854150-N (abstraction,abstract), the selected
synset for the domain concept was SID-05854150-N (abstraction,abstract). This is due to
the fact that the domain concept context available was not enough for the disambiguation
step and intersects with both SID-05854150-N and SID-06468951-N, with an intersection
set containing the same number of elements. A similar behaviour was observed for the
second pair involving Abstract(SUMO) and again for the pair involving the concept
Workshop. For the latter, the same synset was selected for both domain and top-level ontology,
SID-04603081-N (workshop,shop). The concept Abstract appears once more in another
correspondence, for DOLCE, but it was not validated automatically nor manually. For
11 pairs (out the 28), very specific terms are used to express either domain or top-level
concepts, and there are no respective entries in WordNet. As a general purpose resource,
it does not cover as well domain specialized vocabularies. These specialized terms are
usually expressed by noun compounds and represent 39% of correspondences that could
not be validated. These limitations could be addressed by extending the validation to
additional external resources.</p>
      <p>Among the 13 pairs annotated as correct by the evaluators, 5 of them were
validated automatically; 5 others belong to very specific terms for which there are no entries
in WordNet, and 3 were found in WordNet, but not automatically validated: Organization
and Organization (SUMO), Presentation and Proposition (DOLCE), and Topic and
Concept (GFO). For Organization in the domain ontology, the synset SID-08008335-N
(organization,organisation) was selected, which represents a group of people working together,
but for the top-level, a wrong synset was selected, SID-08164585-N (administration,
governance, governing body, establishment, brass, organization, organisation). For the pair
Presentation and Proposition, the context was not enough to identify the correct synset.
While for the domain concept the synset SID-01048697-N (presentation) was selected,
for the top-level the synset SID-06750804-N (proposition) has been chosen and there was
no hierarchical relation between them. For the pair Topic and Concept, the synset
SID06599788-N (subject,topic,theme) and SID-05835747-N (concept,conception,construct)
were selected and, no relation was found between them in WordNet.</p>
      <p>We could also observe that the domain ontology is not terminologically rich
enough (i.e., entities have no labels nor comments) for helping the disambiguation and
selection of the appropriate synsets. In some cases, only the identifier of the concept is
available, once it is a root node in the hierarchy. For instance, the domain concept
Organization, which has 7 WordNet synsets, has no annotations and no neighbours in the
ontology hierarchy. Furthermore, the inconsistencies introduced in the resulting
alignments (merge) should be taken into account. For instance, it seems contradictory that
Conference applicant matches Non physical object and Registered applicant to
Physical object, considered that the latter is a subclass of Conference applicant in the domain
ontology. Finally, we could enrich the generated set of alignments by introducing simple
hierarchical reasoning (if Conference applicant is a Physical object, all its subclasses are
as well). Thus, covering most domain concepts.</p>
    </sec>
    <sec id="sec-10">
      <title>5. Related work</title>
      <p>
        Background knowledge from external resources such as WordNet has been largely
exploited in matching domain ontologies, as a way for improving similarity measures
[
        <xref ref-type="bibr" rid="ref15">Lin and Sandkuhl 2008</xref>
        ]. In [
        <xref ref-type="bibr" rid="ref14">Kwak and Yong 2010</xref>
        ], a lexical similarity for ontology
matching considers the rate of including names of ontology entities and WordNet synset’s
words in an aggregated set (including hypernyms, hyponyms, holonyms and meronyms
relations). Similar to us, they overlap word sets built from the available information on
ontology entities and synsets. However, we exploit the context of ontology entities,
including their sub-concepts, super-concepts and annotations but are limited to hypernym
relations. In [
        <xref ref-type="bibr" rid="ref29">Yatskevich and Giunchiglia 2004</xref>
        ], a set of twelve element-level matchers using
WordNet as background knowledge is proposed. The authors distinguish
knowledgebased matchers (which exploit the structural properties of WordNet, often combined with
statistics from large scale corpora) and gloss-based matchers (which exploit WordNet
synset descriptions). We also exploit the structural properties of WordNet (but focusing
on hypernym relations) and the glosses of synsets. However, they do not exploit the
notion of context of ontology entities. As in our case, both works distinguish equivalence
and subsumption relations. None of the matchers here indicated subsumption (x4.2).
      </p>
      <p>
        A closer approach to ours is proposed in [
        <xref ref-type="bibr" rid="ref25">Schadd and Roos 2012</xref>
        ], where virtual
documents represent the meaning of ontology entities and WordNet entries and entities
are coupled according to their document similarities. A document is constructed from
entities naming and annotation properties and information on the neighbours of the
ontology entity, as we do in this paper. However, we exploit the WordNet hypernym relations
in another way (not for constituting the virtual document associated to a synset), but in
the step of identifying the relation between synsets. As in their approach, ours is
related to the field of Word Sense Disambiguation, which can be carried out using several
different techniques [
        <xref ref-type="bibr" rid="ref22">Navigli 2009</xref>
        ]. Although it is possible to tackle this problem from
various angles and relying on different similarity metrics, here we adopt a naive method
based on the exact match of words in a context. The notion of context has also been
exploited in [
        <xref ref-type="bibr" rid="ref28">Wang 2011</xref>
        ], where semantic description documents (SDD) refer to the
information about concept hierarchies, related properties and instances, or in [
        <xref ref-type="bibr" rid="ref4">David 2011</xref>
        ]
where a bag of words describing a concept is exploited within a mining approach. Other
matching approaches exploiting WordNet include YAM++ [
        <xref ref-type="bibr" rid="ref23">Ngo and Bellahsene 2013</xref>
        ]
and XMAP [
        <xref ref-type="bibr" rid="ref5">Djeddi and Khadir 2014</xref>
        ] systems. YAM++ applies diverse strategies
coming from machine learning, information retrieval and graph matching. For
terminological similarity measures, YAM++ includes Wordnet-based metrics as Lin and Wu-Palmer
[
        <xref ref-type="bibr" rid="ref2">Budanitsky and Hirst 2006</xref>
        ]. XMAP defines a semantic similarity measure, defined using
a domain-specific resource (UMLS) and WordNet. In order to deal with lexical
ambiguity, the notion of scope is introduced, which represents the context where a concept occurs
(i.e., the set of those concepts that are close to a given concept in therms of shortest path
in a semantic network). They exploit super-concepts and sub-concepts in a hierarchy, as
we did, but they do not take into account entity annotations as comments or definitions.
      </p>
      <p>
        Our approach is inspired from those WordNet-based matching and sense
disambiguation approaches for automatically confirming or not generated alignments that rely
on different strategies. There exist different ways to evaluate alignments generated by
matchers [
        <xref ref-type="bibr" rid="ref11">Hong-Hai et al. 2002</xref>
        ]. One classical way consists of comparing alignments
to reference ones (gold standard). However, constructing such references is a
timeconsuming task. They are not always available or may suffer of imperfections or low
coverage. In the lack of such resources, alternatives include manual labeling on
sample alignments, alignment coherence measurements [
        <xref ref-type="bibr" rid="ref19">Meilicke 2011</xref>
        ], and correspondence
patterns mining. Furthermore, the quality of a matcher can be assessed regarding its
suitability for a specific task or application [
        <xref ref-type="bibr" rid="ref12">Isaac et al. 2008</xref>
        ,
        <xref ref-type="bibr" rid="ref10">Hollink et al. 2008</xref>
        ].
Finally, alternative approaches for validating alignments consider the generation of
natural language questions to support end-users in the validation task [
        <xref ref-type="bibr" rid="ref1">Abacha et al. 2014</xref>
        ]
or validation of correspondences using graph-based algorithms in a semi-automatic way
[
        <xref ref-type="bibr" rid="ref27">Serpeloni et al. 2011</xref>
        ].
      </p>
    </sec>
    <sec id="sec-11">
      <title>6. Final remarks and future work</title>
      <p>This paper presented an approach for validating alignments between top-level and domain
ontologies, using WordNet. We presented preliminary results that point to the strengthen
and weaknesses of our approach. In particular, we do not mainly treat terms (but single
words); the appropriate synset may not be selected if a wrong synset overlaps with the
context of a concept, with the same number of elements in the overlapping set; sometimes
the disambiguation process does not help due to the lack of context (cases where the
ontologies have not enough terminological information or are structurally flat); we can
not validate correspondences involving concepts with no entries in the resource; we do
not take into account logical inconsistency nor simple hierarchical reasoning for filtering
out and enhancing the set of correspondences. For overcoming these weaknesses, we plan
to enrich the contexts by adding the contexts of neighbours, use distributional resources
within the disambiguation process for better measuring the overlap of contexts, apply
alternative similarity metrics for measuring the overlap between contexts, and deal with
logical reasoning. We also plan to combine different background resources, including
indirect alignments between WordNet and top-level ontologies and the lexico-semantic
resource BabelNet, which integrates several lexicons and encyclopedias, and explore the
superior layer of concepts in domain ontology to retrieve the more appropriate synset.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <surname>Abacha</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dos</surname>
            <given-names>Reid</given-names>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            , and
            <surname>Mrabet</surname>
          </string-name>
          ,
          <string-name>
            <surname>Y.</surname>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>Question Generation for the Validation of Mapping Adaptation</article-title>
          .
          <source>In 6th International Symposium on Semantic Mining in Biomedicine</source>
          , pages
          <fpage>27</fpage>
          -
          <lpage>31</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          <string-name>
            <surname>Budanitsky</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Hirst</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          (
          <year>2006</year>
          ).
          <article-title>Evaluating WordNet-based Measures of Lexical Semantic Relatedness</article-title>
          .
          <source>Comput. Linguistics</source>
          ,
          <volume>32</volume>
          (
          <issue>1</issue>
          ):
          <fpage>13</fpage>
          -
          <lpage>47</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <string-name>
            <surname>Cheatham</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Hitzler</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <source>Conference v2.0: An Uncertain Version of the OAEI Conference Benchmark. In 13th Semantic Web Conference</source>
          , pages
          <fpage>33</fpage>
          -
          <lpage>48</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          <string-name>
            <surname>David</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>AROMA results for OAEI 2011</article-title>
          .
          <source>In 6th Workshop on Ontology Matching</source>
          , pages
          <fpage>122</fpage>
          -
          <lpage>125</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <surname>Djeddi</surname>
            ,
            <given-names>W. E.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Khadir</surname>
            ,
            <given-names>M. T.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>A Novel Approach Using Context-Based Measure for Matching Large Scale Ontologies</article-title>
          .
          <source>In 16th Conf. on Data Warehousing and Knowledge Discovery</source>
          , pages
          <fpage>320</fpage>
          -
          <lpage>331</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <string-name>
            <surname>Euzenat</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Shvaiko</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2007</year>
          ).
          <source>Ontology Matching. Springer-Verlag, 1st edition.</source>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <string-name>
            <surname>Gangemi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Guarino</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Masolo</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oltramari</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Schneider</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          (
          <year>2002</year>
          ).
          <article-title>Sweetening Ontologies with DOLCE</article-title>
          .
          <source>In 13th Conf. on Knowledge Engineering and Knowledge Management</source>
          , pages
          <fpage>166</fpage>
          -
          <lpage>181</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <string-name>
            <surname>Guarino</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          (
          <year>1998</year>
          ).
          <source>Formal Ontology in Information Systems: Proceedings of the 1st Conference. IOS Press, 1st edition.</source>
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <string-name>
            <surname>Herre</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Heller</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Burek</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hoehndorf</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Loebe</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Michalek</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>2007</year>
          ).
          <article-title>General Formal Ontology (GFO): A Foundational Ontology Integrating Objects and Processes</article-title>
          . In Basic Principles, Research Group Ontologies in Medicine).
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          <string-name>
            <surname>Hollink</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Van Assem</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Isaac</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Schreiber</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          (
          <year>2008</year>
          ).
          <article-title>Two Variations on Ontology Alignment Evaluation: Methodological Issues</article-title>
          .
          <source>In 5th European Semantic Web Conference</source>
          , pages
          <fpage>388</fpage>
          -
          <lpage>401</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          <string-name>
            <surname>Hong-Hai</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Melnik</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Rahm</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          (
          <year>2002</year>
          ).
          <article-title>Comparison of schema matching evaluations</article-title>
          . In Workshop on Web,
          <source>Web-Services, and Database Systems</source>
          , pages
          <fpage>221</fpage>
          -
          <lpage>237</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          <string-name>
            <surname>Isaac</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Matthezing</surname>
          </string-name>
          , H., van der Meij, L.,
          <string-name>
            <surname>Schlobach</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Zinn</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2008</year>
          ).
          <article-title>Putting Ontology Alignment in Context: Usage Scenarios, Deployment and Evaluation in a Library Case</article-title>
          .
          <source>In 5th European Semantic Web Conference</source>
          , pages
          <fpage>402</fpage>
          -
          <lpage>417</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <string-name>
            <surname>Kalfoglou</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Schorlemmer</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2003</year>
          ).
          <article-title>Ontology mapping: the state of the art</article-title>
          .
          <source>Knowledge Engineering Review</source>
          ,
          <volume>18</volume>
          (
          <issue>1</issue>
          ):
          <fpage>1</fpage>
          -
          <lpage>31</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          <string-name>
            <surname>Kwak</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Yong</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>2010</year>
          ).
          <article-title>Ontology Matching based on Hypernym, Hyponym, Holonym and Meronym Sets in WordNet</article-title>
          .
          <source>Web and Semantic Technology</source>
          ,
          <volume>1</volume>
          (
          <issue>2</issue>
          ):
          <fpage>1</fpage>
          -
          <lpage>14</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Sandkuhl</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          (
          <year>2008</year>
          ).
          <article-title>A Survey of Exploiting WordNet in Ontology Matching</article-title>
          .
          <source>In IFIP 20th World Computer Congress</source>
          , pages
          <fpage>341</fpage>
          -
          <lpage>350</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <string-name>
            <surname>Maedche</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Staab</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          (
          <year>2002</year>
          ).
          <article-title>Measuring Similarity between Ontologies</article-title>
          .
          <source>In 13th Conf. on Knowledge Engineering and Knowledge Management</source>
          , pages
          <fpage>251</fpage>
          -
          <lpage>263</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          <string-name>
            <surname>Mascardi</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          , Cord`ı, V., and
          <string-name>
            <surname>Rosso</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2007</year>
          ).
          <article-title>A Comparison of Upper Ontologies</article-title>
          .
          <source>In 8th AI*IA/TABOO Joint Workshop on Agents and Industry</source>
          , pages
          <fpage>55</fpage>
          -
          <lpage>64</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          <string-name>
            <surname>Mascardi</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Locoro</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Rosso</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2010</year>
          ).
          <article-title>Automatic Ontology Matching via Upper Ontologies: A Systematic Evaluation</article-title>
          .
          <source>IEEE Trans. on Knowl. and Data Eng</source>
          .,
          <volume>22</volume>
          (
          <issue>5</issue>
          ):
          <fpage>609</fpage>
          -
          <lpage>623</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <string-name>
            <surname>Meilicke</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Alignment incoherence in ontology matching</article-title>
          .
          <source>PhD thesis</source>
          , University of Mannheim.
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          <string-name>
            <surname>Mika</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oberle</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gangemi</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sabou</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2004</year>
          ).
          <article-title>Foundations for Service Ontologies: Aligning OWL-S to Dolce</article-title>
          .
          <source>In 13th Conf. on World Wide Web</source>
          , pages
          <fpage>563</fpage>
          -
          <lpage>572</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          <string-name>
            <surname>Miller</surname>
            ,
            <given-names>G. A.</given-names>
          </string-name>
          (
          <year>1995</year>
          ).
          <article-title>WordNet: A Lexical Database for English</article-title>
          .
          <source>Commun. ACM</source>
          ,
          <volume>38</volume>
          (
          <issue>11</issue>
          ):
          <fpage>39</fpage>
          -
          <lpage>41</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <string-name>
            <surname>Navigli</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2009</year>
          ).
          <volume>41</volume>
          (
          <issue>2</issue>
          ):
          <volume>10</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>10</lpage>
          :
          <fpage>69</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          <string-name>
            <surname>Ngo</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Bellahsene</surname>
            ,
            <given-names>Z.</given-names>
          </string-name>
          (
          <year>2013</year>
          ). YAM++
          <article-title>- Results for OAEI 2013</article-title>
          .
          <source>In 8th Workshop on Ontology Matching</source>
          , pages
          <fpage>211</fpage>
          -
          <lpage>218</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          <string-name>
            <surname>Niles</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Pease</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          (
          <year>2001</year>
          ).
          <article-title>Towards a Standard Upper Ontology</article-title>
          .
          <source>In Conf. on Formal Ontology in Information Systems</source>
          , pages
          <fpage>2</fpage>
          -
          <lpage>9</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          <string-name>
            <surname>Schadd</surname>
            ,
            <given-names>F. C.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Roos</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          (
          <year>2012</year>
          ).
          <article-title>Coupling of Wordnet Entries for Ontology Mapping Using Virtual Documents</article-title>
          .
          <source>In 7th Workshop on Ontology Matching</source>
          , pages
          <fpage>25</fpage>
          -
          <lpage>36</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          <string-name>
            <surname>Schmidt</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Trojahn</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Vieira</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2016</year>
          ).
          <article-title>Analysing Top-level and Domain Ontology Alignments from Matching Systems</article-title>
          .
          <source>In 11th Workshop on Ontology Matching</source>
          , pages
          <fpage>1</fpage>
          -
          <lpage>12</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          <string-name>
            <surname>Serpeloni</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Moraes</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Bonacin</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Ontology mapping validation</article-title>
          .
          <source>International Journal of Web Portals</source>
          ,
          <volume>3</volume>
          (
          <issue>3</issue>
          ):
          <fpage>1</fpage>
          -
          <lpage>11</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Lily Results on SEALS Platform for OAEI 2011</article-title>
          .
          <source>In 6th Workshop on Ontology Matching</source>
          , pages
          <fpage>156</fpage>
          -
          <lpage>162</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          <string-name>
            <surname>Yatskevich</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Giunchiglia</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          (
          <year>2004</year>
          ).
          <article-title>Element level semantic matching using WordNet</article-title>
          .
          <source>In Meaning Coordination and Negotiation Workshop</source>
          , pages
          <fpage>37</fpage>
          -
          <lpage>48</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>