<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Quality of an Ontology as a Dynamic Optimisation Problem</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Michael Cochez</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Vagan Terziyan</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Mathematical Information Technology University of Jyvaskyla P.</institution>
          <addr-line>O. Box 35 (Agora), 40014, Jyvaskyla</addr-line>
          ,
          <country country="FI">Finland</country>
        </aff>
      </contrib-group>
      <fpage>249</fpage>
      <lpage>256</lpage>
      <abstract>
        <p>The Semantic Web is a proposal from the World Wide Web Consortium aimed at solving problems like data integration and application interoperability. To reach these goals several languages for the representation of semantic data have been proposed. One of the essential concepts behind semantic data is that the data is according to a certain ontology. However, the goals of the semantic web seem challenged because it seems essential for its working that ontologies are agreed upon and shared. This work-in-progress paper describes a rst step to solving these problems. When an ontology is missing or only partially known a system might try to make an approximation of the missing part of the ontology. The quality of this estimated ontology will depend on the context of the application. This paper proposes the solving of a dynamic multi-objective and context sensitive optimisation problem as a way to evalute the quality of the ontology.</p>
      </abstract>
      <kwd-group>
        <kwd>Semantic Web</kwd>
        <kwd>Ontology features</kwd>
        <kwd>Ontology quality</kwd>
        <kwd>Contextual optimisation Key Terms</kwd>
        <kwd>KnowledgeEvolution</kwd>
        <kwd>Intelligence</kwd>
        <kwd>SemanticWebService</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        The World Wide Web Consortium (W3C) regards the Semantic Web as a web
of data. Currently, data is created at extremely high rate and is not available
enough to end users. The Semantic Web aims \To do for machine processable
information (application data) what the World Wide Web has done for
hypertext\ by supporting the creation of interoperable and linked data. [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] Linking
data mostly happens by using shared identi ers and linking concepts by using
shared ontologies.
      </p>
      <p>
        The paper \Which Semantic Web?" [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] gives quite a critical view on many
concepts of the Semantic Web. It states for instance that \Agreeing on a
cataloging scheme for Semantic Web documents is a prerequisite for any sharing of
semantic knowledge. " and \It is easy enough for computers to exchange data about
computational abstractions such as lenames, sizes, usernames, passwords, etc.
It is much harder for computers to exchange information about human-oriented
concepts such as happiness and beauty.". These statements indicate that the
Semantic Web might actually fail in its basic ideas of making decentralised
information management possible. This work-in-progress paper describes the idea
behind one possible approach to overcome these problems.
      </p>
      <p>
        In order to address the rst problem, it would be necessary to create a
system which can make use of semantic data without having knowledge about the
ontology used by the system which generated it. Therefore, it would be needed
to derive the meaning from the data which another application generated. The
approach proposed in this paper is that there would be an optimisation
procedure which yields an ontology for an application processing semantic data.
The second problem is that computer systems might not be suitable to describe
human-oriented concepts. This problem has been tackled in the past by using
fuzzy logic. One approach is described in [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], where a computational model which
maps events and observations to an emotional state uses a fuzzy-logic
representation. This paper is keeping the option of using an ontology based on fuzzy
logic as one possible way of nding an ontology.
      </p>
      <p>
        One important application of the proposed approach can be found in
selfmanaged systems. The `executable reality' approach as described in [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] is one
example of a system which would bene t from the approach described in this
article. `Executable reality' is described in as \an extension of the (Mobile)
Mixed Reality concept". The described extension replaces part of the `static'
retrieval of information by computation of data using context sensitive business
intelligence. When a device with such system is used at a location close by the
sea concepts related to shores, harbours and seashells might become part of the
active ontology. When the device is at a later time point used in a mountainous
area the sea related concepts become partly redundant. If the device has a small
storage capacity, the most optimal ontology will not contain these concepts any
longer. A device which has, on the contrary, an abundant storage capacity but
low processing power should keep the concepts stored to avoid computationally
expensive changes in the ontology.
2
      </p>
    </sec>
    <sec id="sec-2">
      <title>Optimisation</title>
      <p>Optimisation problems are in general statements of problems where a best
solution is to be found according to certain criteria and restrictions. The rst
paragraphs describe a few classes of global optimisation problems in order to
introduce the Context-dependant Dynamic Multi-objective optimisation class in
the last paragraph.</p>
      <p>Static single-objective optimisation is considered a basic type of optimisation
problems. The problem statement consists of a function which will be called f
with domain D and range R. The domain of the function can be given explicitly
as a set or described using constraints on a set. The set used for the range should
have a total order relation 6 de ned on its elements, i.e. there is a transitive,
antisymmetric, and total relation on the elements of R.
De nition 1. An element d 2 D is optimal for a function f , i.e. d 2 opt (f ) ,
8 e 2 D : f (e) 6 f (d)</p>
      <p>The class of optimisation problems described in the previous de nition can be
extended to a multi-objective variant by allowing the range of the function to be a
set of vectors of dimension n. The range of the function f is thus R1 R2 Rn
where we require a (strict) total order relation &lt;i to be de ned on each Ri. The
domain of the function is a set D. Note that all single objective optimisation
problems are multi-objective problems.</p>
      <p>De nition 2. An element d 2 D is multi-objective optimal for a function f ,
i.e. d 2 opt (f ) , 8 e 2 D n fdg : (9 i 2 [1; n] : f (e)i &lt;i f (d)i)</p>
      <p>Dynamic optimisation is essentially not di erent from solving a series of
(multi-objective) static optimisation problems. The dynamism in the series is
to be found in the way the function which is optimised or the constraints on
the domain of the function being optimised are changing. The series can be
represented by a function F , which maps the natural numbers to a set of pairs
of a function and its domain. The functions in the tuples have a domain which
is a subset of the total domain D the optimisation problem is working on. One
way of solving the dynamic optimisation problem is by nding the optimal value
for the functions for all possible values in N and then selecting the maximum
value. We denote opt0(F (t)) to be the optimisation problem with function F (t)1
and constraints F (t)2.</p>
      <p>De nition 3. An element d 2 D is considered optimal for the dynamic
optimisation of the function F : N ! (f unctions; constraints) if 8t 2 N : opt0(F (t))
d.</p>
      <p>Another possible de nition follows:
De nition 4. A function sol is the solution of the dynamic optimisation
problem if sol : N ! D and sol(t) = opt0(F (t))
In other words, a solution is such function which gives the optimal solution for
each possible t 2 N.</p>
      <p>Context-dependant dynamic multi-objective optimisation problems are an
extension of the dynamic optimisation problems de ned in the previous
paragraph. In this case the domain of the function F is a series of what will be called
`contexts' instead of the natural numbers. The solution of the optimisation
problem can be stated in a similar way as the de nitions in the previous paragraphs.
A solution of this type of problem indicates (multi-objective) optimal solutions
in particular contexts, or analog optimal solutions over the range of all possible
contexts.
3</p>
    </sec>
    <sec id="sec-3">
      <title>Features of an Ontology</title>
      <p>
        An application needs schemas or ontologies in order to give meaning to the
semantic data it is processing. Di erent de nitions of ontologies have been
proposed. Gruber [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] de ned, for instance, that \An ontology is an explicit speci
cation of a conceptualization.", which allows for a very broad interpretation. A
more concrete de nition for description of ontologies is OWL2 endorsed by the
World Wide Web Consortium. [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] There is a close correspondence (and
sometimes even compatibility) between ontologies and description logic. OWL2 is for
instance compatible with the description logic SROIQ. [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]
      </p>
      <p>
        For this article, the concrete syntax of the ontology used is not of mayor
importance. More important is the fact that an ontology has certain features.
Examples of features of an ontology include coverage, cohesion and coupling. [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]
In this article the properties which an ontology has independent of any context
will be called the `features' of the ontology. Some literature calls these properties
`quality' factors. The name quality will however be used in one of the following
sections to describe the e ect of features in a context. Other methods for
measuring features of an ontology have been proposed by Burton-Jones et al. [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] and
Yao et al. [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ] and many other feature sets can be found in the literature.
4
      </p>
    </sec>
    <sec id="sec-4">
      <title>Fuzzy Ontologies</title>
      <p>
        For the representation of not exactly de ned sets, fuzzy sets as described in
\Fuzzy Sets" [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] can be used. The elements of a fuzzy set are members of the
set according to a given membership function. This function de nes for each
element of the considered universe a grade of membership in the set. The grade
of membership is a real value in the interval [0; 1], where a value of 0 means that
the element is not in the set and a value of 1 indicates that the element is in the
set. Any value in between indicates up to which extend the element is a member
of the set.
      </p>
      <p>
        Calegari and Giucci used the concept of fuzzy sets to describe what they
call `Fuzzy Ontologies', `Fuzzy Description Logics' and `Fuzzy-OWL'. [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] This
research showed it to be possible to describe ontologies which are not exactly
known by using membership functions. Bobillo and Straccia [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] did a similar
work, but used OWL 2, and proposed a concrete XML syntax for the extension.
5
      </p>
    </sec>
    <sec id="sec-5">
      <title>Ontology Evolution</title>
      <p>
        In research on databases it has been noticed that the schemas which are used
change over time. Ontologies have similar properties. Changes in the domain, the
conceptualisation and the speci cation are unavoidable and the ontology has to
be changed accordingly. The domain changes because the real world changes, the
conceptualisation because the perspective changes and the speci cation changes
when an ontology is to be represented in a language with di erent semantics
and expressiveness. The whole of these changes is called ontology evolution.
Ontologies and database schemas are di erent concepts. Firstly, the ontology
itself can also be part of the data and this way the data becomes self-descriptive.
Secondly, ontologies are explicitly designed for reuse in other context as the initial
context of creation and are, moreover, decentralised by nature. Lastly, ontology
models have, in general, a richer set of properties available for describing the
domain and quite often the border between the schema and instance data is
blurry.[
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]
      </p>
      <p>The same article also describes concrete e ects of ontology evolution on the
data set. For instance removal of a class causes instances to have a less speci c
type, declaring classes disjoint makes instances that are in both classes invalid
and de ning a class as a subclass of another one adds new possible properties
on the subclass.</p>
      <p>
        Despite their di erences, a more recent article by Hartung et al. [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ] claims
that ontology evolution has similar requirements as changes in database schemas
evolution. Ontology evolution requires, for instance, \support for a rich set of
changes, expressive mappings, update propagation to instances and dependant
schemas/ontologies, versioning and user-friendly tools". The same article
compares several approaches for managing and tracking ontology evolution in terms
of the above mentioned criteria.
6
      </p>
    </sec>
    <sec id="sec-6">
      <title>Context and Quality of an Ontology</title>
      <p>In order to talk about the quality of an ontology, one needs to take the context
within which it is used into account. This statement can be supported by an
example. Imagine for instance that one would say that an ontology which contains
more concepts, is better than one with less concepts. This could be true in
certain situations. However, if one imagines a system which only uses the concepts
which are mentioned in the ontology with less concepts, the smaller ontology
might be even better because it uses less memory space.</p>
      <p>
        One way of de ning ontological quality is described in \Data Driven Ontology
Evaluation" [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. This method uses a combination of a corpus and the ontology
to evaluate the quality of the ontology. The corpus is a textual description of the
ontology, which form the basis of di erent approaches of measuring ontologies
described. The paper further elaborates on the fact that there is more as one
quality aspect with regard to ontologies. Factors like price to build, maintenance
and re-use are highlighted as very in uential. Furthermore, the article mentions
that the quality might be subjective to time, location and other contextual
factors.
      </p>
      <p>
        A recent survey on ontology evaluation tools was performed by Aruna et
al. [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ] This survey puts a stress on more technical demands of an ontology in a
working system. Technical properties surveyed are interoperability, turn around
ability, performance, memory allocation, scalability, integration into frameworks
and interfaces. The ontological properties are limited to language conformity
and consistency.
      </p>
      <p>
        Research on improving the quality of an ontology by transformation
operations on an existing ontology is described in [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. The idea is that certain quality
criteria can be ful lled better by a transformation of the existing ontology into a
new, but equally valid ontology. Concrete, several transformations are described
to improve the ontology in terms of homogeneity, totality of properties, stability
over time, and explicitness (as opposed to inferred) and uniformity in
properties. The size of the ontology and simplicity of queries is, according to the same
article, only assessable in a context.
      </p>
      <p>One way of measuring the quality of an ontology might be to compare the
ontology with one or more sets of data which should be described by the ontology.
This comparison can be done using either the open-world assumption which is
typically made in the semantic web or a closed-world assumption.</p>
      <p>Quality does not have to be a one dimensional property. The quality of
an ontology in a context can thus be a multidimensional property. The more
complete the context for optimisation is, the fewer dimensions the quality will
have. A `complete' context will lead to a one dimensional quality.
7</p>
    </sec>
    <sec id="sec-7">
      <title>Quality of Ontology in a Context as a Dynamic</title>
    </sec>
    <sec id="sec-8">
      <title>Multi-objective Optimisation Problem</title>
      <p>As argued in the previous section, it is only reasonable to make statements about
the quality of an ontology in a given context. If for a certain context the quality
for two ontologies is given, then it is possible to make a comparison between the
two ontologies in that particular context. However, quality will not always be
one-dimensional and hence analog to the multi-objective class of optimisation
problems, it is not always possible to say that either the rst or the second
ontology is better.</p>
      <p>When it is possible to compare the quality of ontologies in a context, then it
is also possible to de ne what it means for an ontology to be optimal in a given
context. Because of the fact that there is no total order on the quality of an
ontology in a context, it will be necessary to consider the search for an optimal
ontology for a given context as a multi-objective optimisation problem.</p>
      <p>Let C be the set of contexts, O be the set of ontologies and Q be the set of
qualities. Now we can de ne the optimisation as:
De nition 5. The function sol is the solution of the context-dependant dynamic
multi-objective problem of nding an optimal ontology for a given context ,
sol : C ! O and sol(c) = opt0(F (c))
Where F (C) ! (O ! Q) a function which maps the context c to a function
which incorporates the context when evaluating the quality of an ontology and
its associated domain.</p>
      <p>It makes most sense to use the de nition of context-dependant optimisation
with a function since the system will be used in a real life setting and it is
impossible to predict the optimal ontology over the whole life-time of the system
at any point. It should also be noted that in any real system, the function sol
can only be known partially and most likely by approximation. It is only known
partially because the context is changing all the time and the future contexts
are still unknown. Only an approximation can be found because no real system
can react quick enough to all changes in a complex context in order to compute
a new optimal ontology.</p>
    </sec>
    <sec id="sec-9">
      <title>Future Research Directions</title>
      <p>
        As said in the previous section, it is important to note that the context of the
system in which the ontology is optimised will change dynamically over time.
One important aspect of the context is the history of the system. The reason is
that taking a new ontology into use causes a certain replacement overhead. A
new ontology causing a big overhead has a lower quality in the context of the
system. Cuenca Grau et al. [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ] tried to reduce the cost of consistency checking
of a new ontology by using the previously used ontology, i.e. the history of the
system.
      </p>
      <p>In this article we did not specify any concrete ontology notation to be used
for this type of system. There is a need to evaluate the di erent possible classes
of ontologies and see which properties of ontologies are a ected in this type of
optimisation. Depending on the type of ontology chosen, the system has di erent
options for the development of its ontology. If the ontology would be for instance
a fuzzy ontology, it might even be feasible to change only membership functions
to adapt to changes in the context.</p>
      <p>It is an open question how the system should search for an optimal ontology
for a given context. Furthermore, this article refrained from de ning a concrete
de nition of context and how it should be incorporated in the functions for
optimisation. One popular choice for the incorporation of context is the use of
weighted sum based methods. More research is needed to link particular features
of an ontology to quality aspects, as well as how the context in uences this.</p>
      <p>Next, it seems reasonable to look whether it is possible to notice trends in
the evolution of the ontology. A system could then take the trends into account
when assessing a new ontology or predict resource consumption in the future.</p>
      <p>Lastly, it is interesting to consider what would happen when two separate
systems have their isolated evolutions of the ontology. Questions in that kind of
situation include what should be done to align these ontologies, what to do with
discrepancy between the ontologies, whether these systems can interact if they
are using di erent ontologies, etc. . .
9</p>
    </sec>
    <sec id="sec-10">
      <title>Conclusion</title>
      <p>The aim of this article was to show initial ndings to solve the problems of
interoperability in the Semantic web in case ontologies are not shared among
applications and how to allow these applications to work with more
`humanised' concepts. The rst problem was reduced to a formulation of the nding
of an optimal ontology in a given context in section 7. This context includes
for instance the data processed by the application, the past used ontologies and
constraints related to the system. The challenge of allowing humanised concepts
is attempted by allowing fuzzy logic to be used in this kind of system.</p>
      <p>This article was nancially supported by the `Cloud Software Program' of
TiViT Oy. We would like to thank `Intelligent Precision Solutions and Services
Oy' and in particular Sami Helin for proposing the initial `Cloud Communication
Channel' business case in which this research idea was elaborated.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Carroll</surname>
            ,
            <given-names>J.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Klyne</surname>
          </string-name>
          , G.:
          <article-title>Resource description framework (RDF): Concepts and abstract syntax</article-title>
          .
          <source>W3C recommendation</source>
          ,
          <source>W3C (February</source>
          <year>2004</year>
          ) http://www.w3.org/TR/2004/REC-rdf-concepts-
          <volume>20040210</volume>
          /.
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Marshall</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shipman</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          :
          <article-title>Which semantic web? In: Proceedings of the fourteenth ACM conference on Hypertext and hypermedia</article-title>
          ,
          <source>ACM</source>
          (
          <year>2003</year>
          )
          <volume>57</volume>
          {
          <fpage>66</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>El-Nasr</surname>
            ,
            <given-names>M.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Yen</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ioerger</surname>
            ,
            <given-names>T.R.</given-names>
          </string-name>
          :
          <article-title>Flame|fuzzy logic adaptive model of emotions. Autonomous Agents and Multi-Agent Systems 3 (</article-title>
          <year>2000</year>
          )
          <volume>219</volume>
          {
          <fpage>257</fpage>
          10.1023/A:
          <fpage>1010030809960</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Terziyan</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kaykova</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          :
          <article-title>Towards" executable reality": Business intelligence on top of linked data</article-title>
          .
          <source>In: BUSTECH</source>
          <year>2011</year>
          ,
          <source>The First International Conference on Business Intelligence and Technology</source>
          . (
          <year>2011</year>
          )
          <volume>26</volume>
          {
          <fpage>33</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Gruber</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          , et al.:
          <article-title>A translation approach to portable ontology speci cations</article-title>
          .
          <source>Knowledge acquisition 5(2)</source>
          (
          <year>1993</year>
          )
          <volume>199</volume>
          {
          <fpage>220</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6. Group,
          <string-name>
            <surname>W.O.W.:</surname>
          </string-name>
          <article-title>OWL 2 web ontology language document overview</article-title>
          .
          <source>Technical report, W3C (October</source>
          <year>2009</year>
          ) http://www.w3.org/TR/2009/REC-owl2-overview20091027/.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Patel-Schneider</surname>
            ,
            <given-names>P.F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Motik</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Grau</surname>
            ,
            <given-names>B.C.</given-names>
          </string-name>
          :
          <article-title>OWL 2 web ontology language direct semantics</article-title>
          .
          <source>W3C recommendation</source>
          ,
          <source>W3C (October</source>
          <year>2009</year>
          ) http://www.w3.org/TR/2009/REC-owl2
          <string-name>
            <surname>-</surname>
          </string-name>
          direct-semantics-
          <volume>20091027</volume>
          /.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Ouyang</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zou</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Qu</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhang</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>A method of ontology evaluation based on coverage, cohesion and coupling</article-title>
          . In: FSKD. (
          <year>2011</year>
          )
          <volume>2451</volume>
          {
          <fpage>2455</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Burton-Jones</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Storey</surname>
            ,
            <given-names>V.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sugumaran</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ahluwalia</surname>
            ,
            <given-names>P.:</given-names>
          </string-name>
          <article-title>A semiotic metrics suite for assessing the quality of ontologies</article-title>
          .
          <source>Data Knowl. Eng</source>
          .
          <volume>55</volume>
          (
          <issue>1</issue>
          ) (
          <year>October 2005</year>
          )
          <volume>84</volume>
          {
          <fpage>102</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Yao</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Orme</surname>
            ,
            <given-names>A.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Etzkorn</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>Cohesion Metrics for Ontology Design and Application</article-title>
          .
          <source>Journal of Computer Science</source>
          <volume>1</volume>
          (
          <issue>1</issue>
          ) (
          <year>2005</year>
          )
          <volume>107</volume>
          {
          <fpage>113</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Zadeh</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>Fuzzy sets</article-title>
          .
          <source>Information Control</source>
          <volume>8</volume>
          (
          <year>1965</year>
          )
          <volume>338</volume>
          {
          <fpage>353</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Calegari</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ciucci</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Fuzzy ontology, fuzzy description logics and fuzzy-owl</article-title>
          .
          <source>In: Proceedings of the 7th international workshop on Fuzzy Logic and Applications: Applications of Fuzzy Sets Theory. WILF '07</source>
          , Berlin, Heidelberg, Springer-Verlag (
          <year>2007</year>
          )
          <volume>118</volume>
          {
          <fpage>126</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Bobillo</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Straccia</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          :
          <article-title>Fuzzy ontology representation using owl 2</article-title>
          .
          <source>CoRR abs/1009</source>
          .3391 (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Noy</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Klein</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Ontology evolution: Not the same as schema evolution</article-title>
          .
          <source>Knowledge and Information Systems</source>
          <volume>6</volume>
          (
          <issue>4</issue>
          ) (
          <year>2004</year>
          )
          <volume>428</volume>
          {
          <fpage>440</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Hartung</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Terwilliger</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rahm</surname>
          </string-name>
          , E.:
          <article-title>Recent advances in schema and ontology evolution</article-title>
          .
          <source>Schema Matching and Mapping</source>
          (
          <year>2011</year>
          )
          <volume>149</volume>
          {
          <fpage>190</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Brewster</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Alani</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dasmahapatra</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wilks</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>Data driven ontology evaluation</article-title>
          .
          <source>In: International Conference on Language Resources and Evaluation (LREC</source>
          <year>2004</year>
          ).
          <article-title>(</article-title>
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Aruna</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Saranya</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bhandari</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>A survey on ontology evaluation tools</article-title>
          .
          <source>In: Process Automation, Control and Computing (PACC)</source>
          ,
          <source>2011 International Conference on. (july</source>
          <year>2011</year>
          )
          <volume>1</volume>
          {
          <fpage>5</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Mostow</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fotouhi</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          :
          <article-title>Improving quality of ontology: An ontology transformation approach</article-title>
          .
          <source>In: Data Engineering Workshops</source>
          ,
          <year>2006</year>
          . Proceedings. 22nd International Conference on,
          <source>IEEE</source>
          (
          <year>2006</year>
          )
          <volume>61</volume>
          {
          <fpage>61</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <given-names>Cuenca</given-names>
            <surname>Grau</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            ,
            <surname>Halaschek-Wiener</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            ,
            <surname>Kazakov</surname>
          </string-name>
          ,
          <string-name>
            <surname>Y.</surname>
          </string-name>
          :
          <article-title>History matters: Incremental ontology reasoning using modules</article-title>
          .
          <source>The Semantic Web</source>
          (
          <year>2007</year>
          )
          <volume>183</volume>
          {
          <fpage>196</fpage>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>