<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Scaffolding the Mitochondrial Disease Ontology from extant knowledge sources</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Jennifer D. Warrender</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Phillip Lord</string-name>
          <email>phillip.lord@newcastle.ac.uk</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>School of Computing Science, Newcastle University</institution>
          ,
          <addr-line>Newcastle-upon-Tyne</addr-line>
          ,
          <country country="UK">UK</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2015</year>
      </pub-date>
      <abstract>
        <p>Bio-medical ontologies can contain a large number of concepts. Often many of these concepts are very similar to each other, and similar or identical to concepts found in other bio-medical databases. This presents both a challenge and opportunity: maintaining many similar concepts is tedious and fastidious work, which could be substantially reduced if the data could be derived from pre-existing knowledge sources. In this paper, we describe how we have achieved this for an ontology of the mitochondria using our novel ontology development environment, the Tawny-OWL library.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1 INTRODUCTION</title>
      <p>Bio-medical ontologies vary in size, with largest containing millions
of concepts. Building ontologies of this size is complex,
timeconsuming and expensive and just as challenging to maintain and
update.</p>
      <p>Ontologies are only one of many mechanisms for the
computational representation of knowledge. In some cases,
ontologies are created where many of the needed concepts will be
available elsewhere as terms in different structured representations.
Being able to reuse these representations as a scaffold for the rest
of an ontology might be able to reduce the cost and work-load of
producing ontologies.</p>
      <p>
        This is evidenced by, for instance, SIO
        <xref ref-type="bibr" rid="ref5">(Dumontier et al., 2014)</xref>
        which contains a list of all the chemical elements. Or the Gene
Ontology (GO)
        <xref ref-type="bibr" rid="ref1">(Ashburner et al., 2000)</xref>
        , which contains many terms
related to chemical homeostasis, each of which need to relate to
a specific chemical described in ChEBI
        <xref ref-type="bibr" rid="ref7">(Hastings et al., 2013)</xref>
        .
In addition to being described elsewhere, these concepts are often
highly similar to each other. In extreme cases such as the amino acid
ontology
        <xref ref-type="bibr" rid="ref16 ref9">(Stevens and Lord, 2012)</xref>
        , ontologies can consist of only
related concepts, and “support” concepts that are used to describe
them.
      </p>
      <p>One solution to this is the use of patterns. A pattern is an
abstract specification of an ontology axiomatisation with a number
of “variables”. The pattern is instantiated by providing values for
these variables, which are then expanded into the full axiomatisation
providing one or more concepts.</p>
      <p>
        Patterns have been implemented by a number of different tools,
which differ in how the patterns are specified, and how and when
the values are provided for the variables. For example, termgenie
is a website which allows submission to GO (and others)
        <xref ref-type="bibr" rid="ref4">(Dietze
et al., 2014)</xref>
        . Variable values are entered through a form which then
generates axioms, definitions and cross-references. For instance,
this is the axiomatisation from termgenie when defining the term
“cytosine homeostasis”
should
addressed:
is_a: GO:0048878 {is_inferred="true"}
! chemical homeostasis
intersection_of: GO:0048878
      </p>
      <p>! chemical homeostasis
intersection_of:</p>
      <p>regulates_levels_of CHEBI:16040 ! cytosine
relationship:
regulates_levels_of CHEBI:16040
{is_inferred="true"} ! cytosine</p>
      <p>As well as the axiomatisation, termgenie also generates a
number of different annotations including a definition, submitter
information, and status. With termgenie, patterns are specified
through the use of JavaScript functions.</p>
      <p>
        In addition to termgenie, other systems also allow patterns.
For example, both the desktop and web version of Prote´ge´
contain forms, which grant users the ability to customise the
GUI and specify several axioms at once. In this case, patterns
are declaratively defined (implicitly, with a GUI design) in
XML
        <xref ref-type="bibr" rid="ref17">(Tudorache et al., 2013)</xref>
        . Applications like Populous
        <xref ref-type="bibr" rid="ref10">(Jupp
et al., 2011)</xref>
        and Rightfield
        <xref ref-type="bibr" rid="ref22">(Wolstencroft et al., 2011)</xref>
        use
spreadsheets or spreadsheet-like interfaces to enter data, which
is then transformed into a set of OWL axioms based on a
pattern. In the case of these two, the patterns are specified in
OPPL, a pattern language for OWL which can also be used
independently
        <xref ref-type="bibr" rid="ref6">(Egana Aranguren et al., 2009)</xref>
        . Finally, the Brain
API allows programmatic construction of ontologies in an easy to
use manner using Java
        <xref ref-type="bibr" rid="ref3">(Croset et al., 2013)</xref>
        .
      </p>
      <p>While these systems are all aimed at somewhat different
usecases, they all address the same problem; how to produce a large
number of concepts all of which are similar, and to do so with a
highdegree of repeatability. However, the use of this form of patternised
ontology tool presents a number of problems. These tools provide
a mechanism for adding many axioms at once, but not removing
them again1. If the knowledge changes, then this is a problem as the
axioms added from a given pattern need to be removed or updated.
Furthermore, if the knowledge engineering changes i.e. the pattern
is updated, then all axioms added from any use of the pattern must
also be updated.</p>
      <p>In this paper, we describe how we have addressed these problems
with the Mitochondrial Disease Ontology (MDO), through the use
of the Tawny-OWL environment, which is a fully programmatic
environment for ontology development. With Tawny-OWL, we
can use a pattern-first ontology development process, building
with patterns and data from extant knowledge sources from the
start. This has allowed us to generate a scaffold which we can
then populate further with hand-crafted links between parts of this
scaffold where the knowledge exists. As a result, it is possible to
1 OPPL can remove axioms as well as add them but this is not automatic.
update both the knowledge and the patterns by simply regenerating
the ontology. This process promises to aid in both the construction
and maintenance of ontologies.</p>
      <p>The MDO is available from https://github.com/
jaydchan/tawny-mitochondria. Tawny-OWL is available
from https://github.com/phillord/tawny-owl.
2</p>
    </sec>
    <sec id="sec-2">
      <title>THE MITOCHONDRIA DISEASE ONTOLOGY (MDO)</title>
      <p>
        Mitochondria are complex organelles found in most eukaryotic
cells. Their key function is to enable the production of ATP
through oxidative phosphorylation, providing usable energy for the
rest of the cell. The mitochondria carry their own small genome
containing 37 genes in human. Many other genes are involved in
producing proteins involved in mitochondrial function, but these
are encoded in the nuclear genome. A number of mitochondrial
genes are associated with diseases; the first identified of these is the
MELAS
        <xref ref-type="bibr" rid="ref14">(Pavlakis et al., 1984)</xref>
        , which is most commonly caused by
a point mutation in a tRNA found in the mitochondrial genome.
      </p>
      <p>
        As with many areas of biology, mitochondrial research is a
large, knowledge-rich discipline. Our purpose with the MDO is to
attempt to formalise this knowledge, using an incremental or
“payas-you-go” data integration approach. The ontology here serves
as a tool for reasoning and knowledge exploration, rather than
to form as a reference ontology
        <xref ref-type="bibr" rid="ref15">(Stevens and Lord, 2008)</xref>
        . This
is an approach we have previously found useful in classifying
phosphatases
        <xref ref-type="bibr" rid="ref21">(Wolstencroft et al., 2006)</xref>
        . The hope is that we
can incorporate new knowledge as it is released, checking it for
consistency and cross-linking it with existing knowledge.
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>TAWNY-OWL</title>
      <p>
        In this section, we give a brief description of Tawny-OWL
        <xref ref-type="bibr" rid="ref11 ref19 ref20">(Lord,
2013)</xref>
        and how it supports pattern-first development. Tawny-OWL
is a library written in Clojure, a dialect of lisp. It wraps the
OWL API
        <xref ref-type="bibr" rid="ref10 ref8">(Horridge and Bechhofer, 2011)</xref>
        and allows the fully
programmatic constructions of ontologies. It has a simple syntax
which was modelled on the Manchester Syntax
        <xref ref-type="bibr" rid="ref16 ref9">(Horridge and
PatelSchneider, 2012)</xref>
        , modified to integrate well with Clojure. It can be
used to make simple statements in OWL:
(defclass A :super (some r B))
      </p>
      <p>which makes defines a new class A such that A v 9 r B.
Although this is similar to the equivalent Manchester Syntax
statements, Tawny-OWL provides a feature called “broadcasting”
which is, essentially a form of pattern. So this following statement:
(some r B C)</p>
      <p>is equivalent to the two statements 9 r B and 9 r C. We
apply the first two arguments (some and r) to the remaining ones
consecutively. It also provides simple patterns, such as the covering
axiom, so:
(some-only r B C)</p>
      <p>is equivalent to three statements 9 r B, 9 r C and 8 r (B t C).
While the patterns shown here are provided by Tawny-OWL, end
ontology developers are using the same programmatic environment.
Patterns are encoded as functions and instantiated with function
calls. For instance, we could define some-only as follows:
(defn some-only [property &amp; classes]
(list (some property classes)
(only property</p>
      <p>(or classes))))</p>
      <p>Here defn introduces a new function, property &amp; classes
are the arguments, and list packages the return values as a list.
some, only and or are defined by Tawny-OWL as the appropriate
OWL class constructors.</p>
      <p>
        It is, therefore, possible to build localised patterns — custom
patterns for use predominately with the current ontology
        <xref ref-type="bibr" rid="ref18">(Warrender,
2015)</xref>
        . Patterns can call each other and can be of arbitrary
complexity. The use of Tawny-OWL, therefore, inverts the usual
style of ontology development. Non-patternised classes are just
trivial instantiations of patterns.
4
      </p>
    </sec>
    <sec id="sec-4">
      <title>BUILDING A MITOCHONDRIAL SCAFFOLD</title>
      <p>Following a requirements gathering phase for MDO, it was clear
from our competency questions (for example “What are all the
genes/proteins that are associated with a specific syndrome?”) that
we needed many concepts which were heavily repetitive, and
further which have comprehensive and curated lists available. We
describe these parts of the domain knowledge as the scaffold. For
example, there are around 761 genes whose products are involved
in mitochondrial function. Classes representing these genes do not,
in the first instance, require complex descriptions, and are defined
within MDO as follows:
(defclass Gene)
(defn gene-class [name]
(owl-class name :label name :super Gene))
This pattern is then populated using a simple text file, with
the 761 gene names present. The gene pattern is an extremely
simple pattern, as these concepts are self-standing. Other parts
of the ontology are even simpler; for instance, for describing
mitochondrial anatomy, the classes have similar complexity to the
genes, but there are only 15. In this case, classes are defined with
a pattern and a list “hard-coded” into the MDO source code, rather
than using an external text file. Other patterns are more complex.
For instance, the subclasses of Disease are defined as follows:
(defn disease-class [name omim lname]
(let [disease
(owl-class name
:label name
:super Disease)]
(if-not (nil? omim)
(refine disease
:annotation
(see-also
(str "OMIMID:" omim))))
(if-not (nil? lname)
(refine disease
:label
(str "Long name:" lname)))))
This function adds two annotations to each disease class, if
they are available. This function also demonstrates the use of
conditionals (if), predicates (nil?) and string concatenation
(str); these are not provided by Tawny-OWL, but by Clojure
and demonstrate the value of building Tawny-OWL inside a fully
programmatic environment.
5</p>
    </sec>
    <sec id="sec-5">
      <title>FITTING OUT THE SCAFFOLD</title>
      <p>The top-level of the MDO is shown in Figure 1. Of these classes,
“Paper” and “Term” are described later.</p>
      <p>The remaining classes define the scaffold, which now has a total
of 1357 classes; a break-down of these classes and their sources is
shown in Table 1.</p>
      <p>Class type</p>
      <p>Count</p>
      <p>Data source
Disease 41 UMDF website
Gene 761 The NCBI Gene portal
Human Anatomy 61 The Terminologia Anatomica.
Mitochondrial Anatomy 15 Mitochondrial Research Group
Protein 479 UniProt
Table 1. Table showing the type, number of and data source for each
generic mitochondrial ontology class</p>
      <p>For the next stage of the process, we are now building on top
of this scaffold, using hand-crafted and bespoke knowledge. This
is being achieved by manual extraction of knowledge from papers
about mitochondria. Our initial process is to find references in
papers to the terms that are represented by classes we have built
in the scaffold, and draw explicit relationships between these papers
and the scaffolded knowledge that they describe. Currently, these
classes also use a patternised approach; the raw data is held in a
bespoke (but human readable) syntax2, which is then parsed and
used to instantiate patterns. In total, there are now 2174 classes
2 In this case EDN which is a text representation of Clojure data structures;
it looks rather like JSON.
created from this approach from around 30 papers. These terms
currently are not defined beyond their name and the source paper
from which they were identified. We do not consider them directly
as part of the scaffold, as they are not from an extant knowledge
source, but one that we have created; they are the first layer build
on top of our scaffold. We expect future layers to use the
TawnyOWL syntax directly, as the knowledge increases in complexity and
decreases in regularity.
6</p>
    </sec>
    <sec id="sec-6">
      <title>RESILIANCE TO CHANGE</title>
      <p>One key feature of our development process is that the OWL which
defines the MDO is no longer source code but generated. Rather it is
generated from patterns defined in Tawny-OWL and text files which
are used to instantiate these patterns. The in-memory OWL classes
and associated OWL files are generated on-demand, by evaluating
the patterns. Effectively, we regenerate the ontology every time we
restart the environment. In this section, we consider the types of
changes that can happen, and how these changes impact on MDO.</p>
      <p>The scaffold of MDO is sensitive to changes in its dependency
knowledge sources. First, new terms can be entered into extant
sources, which will necessitate the addition of new classes. For
the MDO, this simply necessitates re-importing the knowledge. The
addition of equivalent new classes will then happen automatically
according to the patterns already defined; no other changes should
be necessary for the MDO, although we may wish to refer to the
new classes in other parts of the ontology.</p>
      <p>
        Second, terms may be removed from dependencies; so, for
example, a disease may be redefined by the UMDF. In many
cases, for the MDO, this is not problematic – the equivalent classes
will simply disappear from the ontology. Tawny-OWL provides
two features to help with changes to terms in the scaffold when
these terms are also referred to outside of the scaffold.
TawnyOWL uses a “declare-before-use” semantics, so removal of classes
from the scaffold will cause fail-fast behaviour when they are
used elsewhere. The Brain environment uses the same semantics
for similar reasons
        <xref ref-type="bibr" rid="ref3">(Croset et al., 2013)</xref>
        . In addition,
TawnyOWL provides a “deprecation” facility which allows the developer
to continue refer to terms from the scaffold which have been
removed, but to receive warnings about this use; this is rather like
obsolescence, but happens automatically3.
      </p>
      <p>Third, the MDO scaffold can also cope straight-forwardly with
changes to patterns. As with the addition or removal of terms
from dependencies, pattern changes will simply take place by
re-evaluating the ontology.</p>
      <p>Finally, the MDO is resilient to changes in ontology engineering
conventions. For example, MDO does not use OBO style numeric
identifiers, nor provide stable IRIs for integration with linked data
sources since these are not critical at the current time4. They,
however, could be added easily to all existing (and future) terms
in a few lines of code, using an existing facility within Tawny-OWL
for minting and persisting numeric identifiers in an automatic, yet
managed, way. This change would just alter IRIs and would have
3 Tawny-OWL is implemented in a Lisp and so is homoiconic; this makes
it particularly straight-forward to automate code updates if we choose.
4 Our initial intention was to use PURLS from www.purl.org but have
found practical problems with generating these.
no impact on references between concepts inside or outside of the
scaffold.</p>
      <p>In conclusion, as well as enabling rapid construction of the MDO,
we believe that the pattern-first scaffolding approach should also
allow easy maintenance of the ontology.
7</p>
    </sec>
    <sec id="sec-7">
      <title>DISCUSSION</title>
      <p>In this paper, we have described how we have used a number of
extant knowledge sources, combined with patterns defined using the
Tawny-OWL library to rapidly, reliably and repeatedly construct a
scaffold for MDO.</p>
      <p>
        We have previously used a related patternised methodology
to construct a complex ontology describing human chromosome
rearrangements (i.e. The Karyotype Ontology (KO)
        <xref ref-type="bibr" rid="ref11 ref19 ref20">(Warrender and
Lord, 2013b)</xref>
        ). However, unlike KO, the mitochondrial knowledge
we want to encapsulate is found in numerous independent sources
(e.g. published papers and online databases) and in a variety
of formats (e.g. “free text” and CSV); the use of several
patterns to form a scaffold is unique to MDO. Conversely, the
axiomatisation of MDO from these sources is simple; this cannot
be said for KO, most of which is generated from a single
large pattern
        <xref ref-type="bibr" rid="ref11 ref19 ref20">(Warrender and Lord, 2013a)</xref>
        . In addition, while
our knowledge of the karyotype is constrained and is essentially
finished, the community’s understanding of mitochondria and
mitochondrial disease is incomplete and will grow in response to
the demands of changing knowledge.
      </p>
      <p>This methodology is extremely attractive for a number of reasons.
First of all, it allows a very rapid way of scaffolding an ontology
for a complex area of knowledge. At this stage, most of the classes
created are simple and self-standing, although in some cases do have
relationships to other entities in the scaffold. At this point, we have
built the ontological equivalent of a data warehouse: terms have
been taken from elsewhere and have undergone a form of schema
reconciliation into ontological classes.</p>
      <p>
        One key feature of the MDO is that it has been built using
tools designed for software development; these tools are relatively
advanced and well-maintained5
        <xref ref-type="bibr" rid="ref11 ref19 ref20">(Lord, 2013)</xref>
        . Moreover, recreating
the MDO ontology from our original Tawny-OWL source code is
an intrinsic part of the development process; there is no complex
release process and any ontology developer can recreate the OWL
file with a single command. While, the system as it stands has
a high-degree of replicability, the design decisions implicit in the
source code are not necessarily apparent. For the basic scaffold this
is, perhaps, not a major issue, however as MDO is developed outside
of its scaffold , we expect to integrate more documentation into the
source code itself, using lentic, a recently developed tool for literate
programming
        <xref ref-type="bibr" rid="ref12">(Lord, 2015)</xref>
        .
      </p>
      <p>
        We believe that the engineering process that we have used to
build the scaffold is resilient to change, as described in Section 6.
Despite this resilience, our use of external sources of knowledge
does bring with it new dependencies, with all the issues that this
entails for change management. We believe that we can manage this
by borrowing best practice from software engineering. Importing
knowledge into the scaffold can, in many cases, happens entirely
automatically from our extant knowledge sources. Considering just
5 And, usefully, not dependent on academic developers for future
maintenance.
the gene lists, we can either import from a local, fixed copy of
this list, or take the current version live from the NCBI portal. In
software engineering terms, the former is a release dependency
and provides stability, while the latter is a snapshot dependency
which will fail-fast, allowing rapid incorporation of new knowledge.
The latter is particularly useful within a continuous integration
environment which are used with other ontologies
        <xref ref-type="bibr" rid="ref13">(Mungall et al.,
2012)</xref>
        , and are also fully supported by Tawny-OWL
        <xref ref-type="bibr" rid="ref11 ref19 ref20">(Lord, 2013)</xref>
        .
      </p>
      <p>
        Although we have not described its usage here, with the MDO we
are not forced to use Tawny-OWL for all development. It would be
possible to combine predominately hand-crafted development using
Prote´ge´, for instance, with some patternised classes; for example,
the OBI uses this approach
        <xref ref-type="bibr" rid="ref2">(Brinkman et al., 2010)</xref>
        . For, the MDO,
in fact almost all terms other than the top-level has been created
from other syntaxes, generally a flat-file. For larger projects, we
envisage that most ontology developers would not need to use
the programmatic nature of Tawny-OWL. While we appreciate the
value of a single environment, a tool should not force all users into
it.
      </p>
      <p>In this paper, we have described our approach to building the
MDO using a patternised scaffold based around existing knowledge
sources. While the work described in this paper allows us to
integrate structured data into an ontology, we are now investigating
new ways of integrating unstructured literate-based knowledge into
our ontology; while we have started the process of formalising,
this new knowledge is far from finished. As described in this
paper, though, a pattern-first, scaffolded approach to ontology
development has enabled us to make significant advances with the
MDO. We believe that this approach is likely to be applicable to
many other domains also.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <surname>Ashburner</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ball</surname>
            ,
            <given-names>C. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Blake</surname>
            ,
            <given-names>J. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Botstein</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Butler</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cherry</surname>
            ,
            <given-names>J. M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Davis</surname>
            ,
            <given-names>A. P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dolinski</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dwight</surname>
            ,
            <given-names>S. S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Eppig</surname>
            ,
            <given-names>J. T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Harris</surname>
            ,
            <given-names>M. A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hill</surname>
            ,
            <given-names>D. P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>IsselTarver</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kasarskis</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lewis</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Matese</surname>
            ,
            <given-names>J. C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Richardson</surname>
            ,
            <given-names>J. E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ringwald</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rubin</surname>
            ,
            <given-names>G. M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sherlock</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          (
          <year>2000</year>
          ).
          <article-title>Gene ontology: tool for the unification of biology. The Gene Ontology Consortium</article-title>
          .
          <source>Nature Genetics</source>
          ,
          <volume>25</volume>
          (
          <issue>1</issue>
          ),
          <fpage>25</fpage>
          -
          <lpage>29</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          <string-name>
            <surname>Brinkman</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Courtot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Derom</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fostel</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>He</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lord</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Malone</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parkinson</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Peters</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rocca-Serra</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ruttenberg</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sansone</surname>
            ,
            <given-names>S.-A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Soldatova</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Stoeckert</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Turner</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zheng</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          , and the OBI consortium (
          <year>2010</year>
          ).
          <article-title>Modeling biomedical experimental processes with obi</article-title>
          .
          <source>Journal of Biomedical Semantics</source>
          ,
          <volume>1</volume>
          (
          <issue>Suppl 1</issue>
          ),
          <fpage>S7</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <string-name>
            <surname>Croset</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Overington</surname>
            ,
            <given-names>J. P.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Rebholz-Schuhmann</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>Brain: biomedical knowledge manipulation</article-title>
          .
          <source>Bioinformatics</source>
          ,
          <volume>29</volume>
          (
          <issue>9</issue>
          ),
          <fpage>1238</fpage>
          -
          <lpage>1239</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          <string-name>
            <surname>Dietze</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Berardini</surname>
            ,
            <given-names>T. Z.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Foulger</surname>
            ,
            <given-names>R. E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hill</surname>
            ,
            <given-names>D. P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lomax</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>OsumiSutherland</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Roncaglia</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Mungall</surname>
            ,
            <given-names>C. J.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>Termgenie - a web application for pattern-based ontology class generation</article-title>
          .
          <source>Journal of Biomedical Semantics</source>
          ,
          <volume>5</volume>
          (
          <issue>1</issue>
          ),
          <fpage>48</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <surname>Dumontier</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Baker</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Baran</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Callahan</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chepelev</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Toledo</surname>
            ,
            <given-names>J. C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Del Rio</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Duck</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Furlong</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Keath</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Klassen</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>McCusker</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rosinach</surname>
            ,
            <given-names>N. Q.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Samwald</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rosales</surname>
            ,
            <given-names>N. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wilkinson</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Hoehndorf</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>The Semanticscience Integrated Ontology (SIO) for biomedical research and knowledge discovery</article-title>
          .
          <source>Journal of Biomedical Semantics</source>
          ,
          <volume>5</volume>
          (
          <issue>1</issue>
          ),
          <volume>14</volume>
          +.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <string-name>
            <given-names>Egana</given-names>
            <surname>Aranguren</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Stevens</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            , and
            <surname>Antezana</surname>
          </string-name>
          ,
          <string-name>
            <surname>E.</surname>
          </string-name>
          (
          <year>2009</year>
          ).
          <article-title>Transforming the axiomisation of ontologies: The ontology pre-processor language</article-title>
          .
          <source>Nature Precedings.</source>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <string-name>
            <surname>Hastings</surname>
            , J., de Matos,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dekker</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ennis</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Harsha</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kale</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Muthukrishnan</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Owen</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Turner</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Williams</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Steinbeck</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>The ChEBI reference database and ontology for biologically relevant chemistry: enhancements for 2013</article-title>
          .
          <source>Nucleic Acids Research</source>
          ,
          <volume>41</volume>
          (
          <issue>D1</issue>
          ),
          <fpage>D456</fpage>
          -
          <lpage>D463</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Bechhofer</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>The OWL API: A Java API for OWL ontologies</article-title>
          .
          <source>Semant. web</source>
          ,
          <volume>2</volume>
          (
          <issue>1</issue>
          ),
          <fpage>11</fpage>
          -
          <lpage>21</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Patel-Schneider</surname>
            ,
            <given-names>P. F.</given-names>
          </string-name>
          (
          <year>2012</year>
          ).
          <article-title>Owl 2 web ontology language manchester syntax (second edition)</article-title>
          .
          <source>Technical report.</source>
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          <string-name>
            <surname>Jupp</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Iannone</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Klein</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Owen</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Schanstra</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wolstencroft</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Stevens</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>Populous: a tool for building owl ontologies from templates</article-title>
          .
          <source>BMC Bioinformatics</source>
          ,
          <volume>13</volume>
          (
          <issue>Suppl 1</issue>
          ),
          <fpage>S5</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          <string-name>
            <surname>Lord</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>The Semantic Web takes Wing: Programming Ontologies with TawnyOWL</article-title>
          . http://arxiv.org/abs/1303.0213.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          <string-name>
            <surname>Lord</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>Lenticular text: Looking at code from different angles</article-title>
          . http:// www.russet.org.uk/blog/3035.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <string-name>
            <surname>Mungall</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dietze</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Carbon</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ireland</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bauer</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Lewis</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          (
          <year>2012</year>
          ).
          <article-title>Continuous integration of open biological ontology libraries</article-title>
          . http: //bio-ontologies.
          <source>knowledgeblog.org/405.</source>
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          <string-name>
            <surname>Pavlakis</surname>
            ,
            <given-names>S. G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Phillips</surname>
            ,
            <given-names>P. C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>DiMauro</surname>
          </string-name>
          , S.,
          <string-name>
            <surname>De Vivo</surname>
            ,
            <given-names>D. C.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Rowland</surname>
            ,
            <given-names>L. P.</given-names>
          </string-name>
          (
          <year>1984</year>
          ).
          <article-title>Mitochondrial myopathy, encephalopathy, lactic acidosis, and strokelike episodes: a distinctive clinical syndrome</article-title>
          . Ann. Neurol.,
          <volume>16</volume>
          (
          <issue>4</issue>
          ),
          <fpage>481</fpage>
          -
          <lpage>488</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          <string-name>
            <surname>Stevens</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          and Lord,
          <string-name>
            <surname>P.</surname>
          </string-name>
          (
          <year>2008</year>
          ).
          <article-title>Application of ontologies in bioinformatics</article-title>
          . In S. Staab and R. Studer, editors,
          <source>Handbook on Ontologies in Information Systems</source>
          . Springer, second edition.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <string-name>
            <surname>Stevens</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          and Lord,
          <string-name>
            <surname>P.</surname>
          </string-name>
          (
          <year>2012</year>
          ).
          <article-title>Semantic publishing of knowledge about amino acids</article-title>
          . http://ceur-ws.
          <source>org/</source>
          Vol-
          <volume>903</volume>
          /paper-06.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          <string-name>
            <surname>Tudorache</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nyulas</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Noy</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Musen</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2013</year>
          ).
          <article-title>Using semantic web in icd-11: Three years down the road</article-title>
          . In H. Alani,
          <string-name>
            <given-names>L.</given-names>
            <surname>Kagal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Fokoue</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Groth</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Biemann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Parreira</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Aroyo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Noy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Welty</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          K. Janowicz, editors,
          <source>The Semantic Web ISWC</source>
          <year>2013</year>
          , volume
          <volume>8219</volume>
          of Lecture Notes in Computer Science, pages
          <fpage>195</fpage>
          -
          <lpage>211</lpage>
          . Springer Berlin Heidelberg.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          <string-name>
            <surname>Warrender</surname>
            ,
            <given-names>J. D.</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>The Consistent Representation of Scientific Knowledge: Investigations into the Ontology of Karyotypes and Mitochondria</article-title>
          .
          <source>Ph.D. thesis</source>
          , School of Computing Science, Newcastle University.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <string-name>
            <surname>Warrender</surname>
            ,
            <given-names>J. D.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Lord</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2013a</year>
          ).
          <article-title>A pattern-driven approach to biomedical ontology engineering</article-title>
          .
          <source>SWAT4LS</source>
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          <string-name>
            <surname>Warrender</surname>
            ,
            <given-names>J. D.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Lord</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2013b</year>
          ).
          <article-title>The Karyotype Ontology: a computational representation for human cytogenetic patterns</article-title>
          .
          <source>Bio-Ontologies SIG</source>
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          <string-name>
            <surname>Wolstencroft</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lord</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tabernero</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brass</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Stevens</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2006</year>
          ).
          <article-title>Protein classification using ontology classification</article-title>
          .
          <source>Bioinformatics</source>
          ,
          <volume>22</volume>
          (
          <issue>14</issue>
          ),
          <fpage>e530</fpage>
          -
          <lpage>538</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <string-name>
            <surname>Wolstencroft</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Owen</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Krebs</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mueller</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Snoep</surname>
            ,
            <given-names>J. L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>du Preez</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Goble</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2011</year>
          ).
          <article-title>RightField: embedding ontology annotation in spreadsheets</article-title>
          .
          <source>Bioinformatics</source>
          ,
          <volume>27</volume>
          (
          <issue>14</issue>
          ),
          <fpage>2021</fpage>
          -
          <lpage>2022</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>