<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Malaria study data integration and information retrieval based on OBO Foundry ontologies</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Jie Zheng</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
          <xref ref-type="aff" rid="aff3">3</xref>
          <xref ref-type="aff" rid="aff4">4</xref>
          <xref ref-type="aff" rid="aff5">5</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>JaShon Cade</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
          <xref ref-type="aff" rid="aff3">3</xref>
          <xref ref-type="aff" rid="aff4">4</xref>
          <xref ref-type="aff" rid="aff5">5</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Brian Brunk</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
          <xref ref-type="aff" rid="aff3">3</xref>
          <xref ref-type="aff" rid="aff4">4</xref>
          <xref ref-type="aff" rid="aff5">5</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>David S. Roos</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
          <xref ref-type="aff" rid="aff3">3</xref>
          <xref ref-type="aff" rid="aff4">4</xref>
          <xref ref-type="aff" rid="aff5">5</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Christian J. Stoeckert Jr.</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
          <xref ref-type="aff" rid="aff3">3</xref>
          <xref ref-type="aff" rid="aff4">4</xref>
          <xref ref-type="aff" rid="aff5">5</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Steven A. Sullivan</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
          <xref ref-type="aff" rid="aff3">3</xref>
          <xref ref-type="aff" rid="aff4">4</xref>
          <xref ref-type="aff" rid="aff5">5</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Jane M. Carlton</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
          <xref ref-type="aff" rid="aff3">3</xref>
          <xref ref-type="aff" rid="aff4">4</xref>
          <xref ref-type="aff" rid="aff5">5</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Bryan Greenhouse, Grant Dorsey Department of Medicine University of California San Francisco San Francisco</institution>
          ,
          <addr-line>CA</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Center for Genomics &amp; Systems Biology Department of Biology New York University New York</institution>
          ,
          <addr-line>NY</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>EuPath Bioinformatics Resource Center University of Pennsylvania Philadelphia</institution>
          ,
          <addr-line>PA</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff3">
          <label>3</label>
          <institution>Gabriel Carrasco-Escobar, Dionicia Gamboa Universidad Peruana Cayetano Heredia Lima</institution>
          ,
          <country country="PE">Peru</country>
        </aff>
        <aff id="aff4">
          <label>4</label>
          <institution>Paula Maguina-Mercedes, Joseph M. Vinetz Division of Infectious Diseases University of California San Diego La Jolla</institution>
          ,
          <addr-line>CA</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff5">
          <label>5</label>
          <institution>San Emmanuel James, Emmanuel Arinaitwe Infectious Diseases Research Collaboration Kampala</institution>
          ,
          <country country="UG">Uganda</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>- The International Centers of Excellence in Malaria Research (ICEMR) projects involve studies to understand the epidemiology and transmission patterns of malaria in different geographic regions. Two major challenges of integrating data across these projects are: (1) standardization of highly heterogeneous epidemiologic data collected by various ICEMR projects; (2) provision of user-friendly search strategies to identify and retrieve information of interest from the very complex ICEMR data. We pursued an ontology-based strategy to address these challenges. We utilized and contributed to the Open Biological and Biomedical Ontologies to generate a consistent semantic representation of three different ICEMR data dictionaries that included ontology term mappings to data fields and allowed values. This semantic representation of ICEMR data served to guide data loading into a relational database and presentation of the data on web pages in the form of search filters that reveal relationships specified in the ontology and the structure of the underlying data. This effort resulted in the ability to use a common logic for storing and display of data on study participants, their clinical visits, and epidemiological information on their living conditions (dwelling) and geographic location. Users of the Plasmodium Genomics Resource, PlasmoDB, accessing the ICEMR data will be able to search for participants based on environmental factors such as type of dwelling, location or mosquito biting rate, characteristics such as age at enrollment, relevant genotypes or gender and visit data such as laboratory findings, diagnoses, malaria medications, symptoms, and other factors.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>
        The ICEMR program is a global network of 10 independent
research centers created to improve understanding of the
epidemiology and transmission patterns of malaria in different
geographic regions [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Integrating data generated by these
Centers into the Plasmodium Genomics Resource (PlasmoDB)
[
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], a component of the Eukaryotic Pathogen Bioinformatics
Resource Center (EuPath BRC), provides web-enabled access
to ICEMR project members, and ultimately the broader
international research community. Common data collected
across all ICEMR projects are represented in Figure 1.
However, data produced by the various ICEMR projects is
heterogeneous with respect to origin, type of data, format, and
spatio-temporal scale. The main challenges of sharing and
integration of ICEMR data include standardizing the complex
and heterogeneous data for consistent representation and
providing a user-friendly interface for easy exploration of the
data for constructing searches..
      </p>
      <p>
        Ontologies play a crucial role in heterogeneous data
integration by supporting consistent data representation and
providing a semantic framework to reveal the relationships
between data thereby facilitating information retrieval and new
knowledge discovery [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. We made use of the Open Biological
and Biomedical Ontologies (OBO) Foundry [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] which
promotes interoperable ontologies and provides a listing of
ontologies seeking to follow Foundry principles. These
ontologies were used to provide a common understanding of
what the information collected according to different ICEMR
data dictionaries and case record forms was about. The
OBObased mappings were useful for guiding data loading and
queries but were not directly usable for providing intuitive
display of the available data on search forms. These were
combined in a EuPath application ontology. Using WebProtege
[5], we created an ICEMR terminology to organize the classes
of data, create top-level categories, and re-label terms
according to user preference while still maintaining the OBO
IRIs where applicable to preserve the semantic underpinnings.
The result was a linked OBO-based application ontology and
web display terminology to provide interoperability and
intuitive access to the datasets based on different data
dictionaries.
      </p>
    </sec>
    <sec id="sec-2">
      <title>II. METHODS</title>
      <sec id="sec-2-1">
        <title>A. ICEMR data and data dictionaries</title>
        <p>Multiple ICEMR projects have provided data for inclusion
in PlasmoDB. Each ICEMR project has provided data
dictionaries covering all data variables and values required for
interpreting the associated data. By data dictionary, we mean
a list of terms with definitions and specification of data
variables, data types, format of data, and allowed values
(including controlled vocabulary values). Data dictionaries are
used in data exchanges among ICEMR projects and sharing
with different repositories. However, data dictionaries from the
different ICEMR projects generally look very different from
each other in terms of type and quantity of content.</p>
      </sec>
      <sec id="sec-2-2">
        <title>B. Consistent representation of ICEMR data</title>
        <p>
          To standardize the data dictionaries from different ICEMR
projects, the variables and controlled vocabulary values were
mapped to OBO ontologies. These included the Ontology for
Biomedical Investigations (OBI) [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ], Phenotype qualities
(PATO) [7], Ontology for General Medical Science (OGMS)
[8], Environmental Ontology (EnVO) [9], Disease Ontology
(DO) [
          <xref ref-type="bibr" rid="ref7">10</xref>
          ], Drug Ontology (DRON) [
          <xref ref-type="bibr" rid="ref8">11</xref>
          ], Infectious Disease
Ontology (IDO) [
          <xref ref-type="bibr" rid="ref9">12</xref>
          ], Human Phenotype Ontology (HP) [
          <xref ref-type="bibr" rid="ref10">13</xref>
          ],
Information Artifact Ontology (IAO) [
          <xref ref-type="bibr" rid="ref11">14</xref>
          ], Ontology for
Biobanking (OBIB) [
          <xref ref-type="bibr" rid="ref12">15</xref>
          ], and Symptom Ontology (SYMP)
[
          <xref ref-type="bibr" rid="ref13">16</xref>
          ]. The mapping of terms specified in the data dictionaries to
OBO ontologies was performed using the BioPortal annotator
web services [
          <xref ref-type="bibr" rid="ref14">17</xref>
          ]. The annotator service can accurately
(&gt;95%) tag text with ontology terms. However, ontologies in
the annotator might not be the latest version since these need to
go through an indexing process before being added to the
annotator. For terms where mappings were not found using the
        </p>
        <p>
          Supported in part by National Institute of Allergy and Infectious Diseases
National Institutes of Health, Department of Health and Human Services
Contract No. HHSN272201400030C, U.S. Public Health Service cooperative
agreements U19AI089674 (MGD) and U19AI089681 (JMV).
annotator, the BioPortal search web services [
          <xref ref-type="bibr" rid="ref15">18</xref>
          ] were used.
Both annotator and search results were reviewed manually.
        </p>
        <p>Consistent representation of ICEMR data was achieved
once the variables and values in the different ICEMR data
dictionaries were either mapped to existing ontology terms or
new ontology terms were created for that purpose. New
ontology terms were created using two approaches.</p>
        <p>a) If the terms were general and in a domain which have
been covered by an OBO ontology, they were submitted to the
relevant ontology via its issue tracker to be added in by the
ontology developers. For example, disease terms were
submitted to the DO tracker and terms related to the
environment were submitted to the EnVO tracker.</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>b) If the terms were specific to the ICEMR projects,</title>
      <p>they were added in the Eupath ontology. The Eupath ontology
is an application ontology developed for providing terms to
annotate data in the EuPath BRC. The EuPath ontology was
built based on OBI with integration of other OBO ontologies
such as PATO, OGMS, DO, etc. when needed.</p>
      <sec id="sec-3-1">
        <title>C. Organization of ICEMR data dictionary variables for guiding searches of ICEMR data</title>
        <p>The ontological mapping of data dictionary variables
provides semantic clarity of types. However, organization
according to term types (e.g., processes, material entities,
qualities, etc.) does not necessarily provide intuitive listing on
web sites for mining the data. As illustrated in Figure 1, the
five main types of interest are ‘participants’, ‘dwellings’,
(clinical) ‘visits’, ‘entomological measurements’ and
‘geographic location’. Therefore, we organized the data
dictionary variables into categories based on their relation to
these types. Within each category, the data dictionary variables
are grouped based on the mapped OBO ontology terms. For
example, ‘height’, ‘weight’, and ‘temperature’ (measurement
data) are grouped together in the ‘physical examination’
category (which in turn is placed in the ‘visit’ category). The
outcome of categorization of the variables from the multiple
ICEMR data dictionaries is the ICEMR terminology and is the
basis for displaying search parameters of this data on the
PlasmoDB website. The ICEMR terminology is represented in
the OWL format containing only ‘is a’ relations enabling
visualization of the ICEMR data dictionary hierarchy
organization using ontology editors. WebProtege [5] is a
webbased collaborative ontology development platform and
provides a means for domain experts to review and post
comments on terms. We uploaded the ICEMR terminology to
WebProtege and used it for collaboratively reviewing both the
organization of the ICEMR terminology and the labels of terms
to be displayed on the PlasmoDB web site before loading the
ICEMR data into the database. This approach ensured that the
data was correctly displayed on PlasmoDB for each ICEMR
project. For the ICEMR terminology, we specified display
labels using the rdfs:label annotation property as they are the
default term labels rendered on WebProtege. In addition, we
used annotation properties to specify ontological names,
definitions, whether the term was an organizing category or a
variable. If the term corresponded to a data dictionary variable,
then annotation properties were also used for the original
variable name in the data dictionary and source, the mapped
ontology term, and the ontological definition. The common
display labels in the ICEMR terminology were agreed upon by
the contributing ICEMR projects. Each contributing ICEMR
project had variables unique to that project. Therefore, the
application of the ICEMR terminology for organization of each
ICEMR data dictionary resulted in different but still consistent
outputs. The application of the ICEMR terminologies to the
different projects can be viewed at the WebProtege site
(http://webprotege.stanford.edu/) as “ICEMR Amazonia”,
“ICEMR Indian”, and “ICEMR PRISM” (Uganda ICEMR
project).</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>III. RESULTS</title>
      <sec id="sec-4-1">
        <title>A. ICEMR data and data dictionaries</title>
        <p>Longitudinal data from three ICEMR projects with studies
in Uganda, India, and Amazonia were submitted for inclusion
in PlasmoDB. Data and data dictionaries from the Uganda and
Indian ICEMR projects were provided in English whereas data
and the data dictionary from the Amazonia ICEMR project
were in Spanish. The Amazonia ICEMR project also provided
a translated data dictionary in English. All three ICEMR
projects provided participant data, dwelling data on
participants, and participant-associated clinical visit data. The
Uganda ICEMR project also submitted entomological
measurement data.</p>
        <p>The Amazonia ICEMR data dictionary included 84
variables and 179 controlled values for 26 variables. The
Indian ICEMR data dictionary contained 118 variables with
149 controlled values for 32 variables. The Uganda ICEMR
data dictionary contained 121 different kinds of variables and
481 controlled values for 21 variables.</p>
      </sec>
      <sec id="sec-4-2">
        <title>B. Ontology term mapping</title>
        <p>Variables and values specified in the ICEMR data
dictionaries were mapped to 10 different OBO Foundry
ontologies (listed in the Methods). Table 1 lists the mapping
results for each ICEMR project. A total of 209 new terms were
added to the EuPath ontology for unmapped ICEMR variables.
The EuPath ontology can be viewed on the WebProtege site
(http://webprotege.stanford.edu/) as the “EuPath ontology”
project.</p>
        <p>Data dictionary variables from the different ICEMR
projects referring to the same thing were often different. For
example, “edad” in the Amazonia ICEMR data dictionary,
“age_en” in the Indian ICEMR data dictionary, and “age” in
the Uganda ICEMR data dictionary all refer to participant age
at the time of enrollment and mapped to the ontology term
EUPATH_0000120: ‘age since birth at time of enrollment’. As
another example of the encountered heterogeneity, Table 2
shows a sampling of mapping between symptom related
variables to ontology terms.</p>
        <p>Ontology term mapping was also performed on the
controlled values of variables. 413 controlled values used in
the Uganda ICEMR data were mapped to OBO ontology terms.
The remaining 68 unmapped terms were added into the EuPath
ontology. Few corresponding ontology terms were found for
the controlled values in the Amazonia and Indian ICEMR data
(14 for Amazonia and 5 for Indian, respectively). For those
values without mapped ontology terms, we have created
standardized labels and will add the terms to either OBO
ontologies or EuPath ontology as described in the Methods.</p>
        <p>After ontology term mapping and standardization of value
labels across data from multiple ICEMR projects, we generated
(data dictionary to standardized) term mapping files for each
ICEMR. These mapping files were used in the ICEMR project
data loading process and enabled consistent data representation
in the PlasmoDB database.
dDiacttaionary Ontology term ID Ontology term label InCamEMeR display
abdominalpain HP_0002027 Abdominal pain Abdominal pain
apainduration EUPATH_0000154 duration of abdominal Abdominal pain
pain duration
Anorexia SYMP_0000523 anorexia Anorexia
aduration EUPATH_0000155 duration of anorexia Anorexia duration
Cough SYMP_0000614 cough Cough
cduration EUPATH_0000156 duration of cough Cough duration
Diarrhea DOID_13250 diarrhea Diarrhea
dduration EUPATH_0000157 duration of diarrhea Diarrhea duration
Fatigue SYMP_0019177 fatigue Fatigue
fmduration EUPATH_0000158 duration of fatigue Fatigue duration
febrile EUPATH_0000097 febrile Febrile
fever EUPATH_0000100 subjective fever Fever (subjective)
Headache HP_0002315 Headache Headache
hduration EUPATH_0000159 duration of headache Headache duration
Jaundice HP_0000952 Jaundice Jaundice
jduration EUPATH_0000160 duration of jaundice Jaundice duration
jointpains SYMP_0000064 joint pain Joint pains
djointpains EUPATH_0000161 duration of joint pains Joint pains
duration
muscleaches EUPATH_0000252 Muscle aches Muscle aches
mduration EUPATH_0000162 duration of muscle Muscle aches
aches duration
rfa OGMS_0000015 clinical history Other medical
complaint
seizure SYMP_0000124 seizure Seizures
sduration EUPATH_0000163 duration of seizures Seizures duration
fduration EUPATH_0000164 duration of subjective Subjective fever
fever duration
Vomiting HP_0002013 Vomiting Vomiting
vduration EUPATH_0000165 duration of vomiting Vomiting duration</p>
      </sec>
      <sec id="sec-4-3">
        <title>C. Organization of terms for search filters and exploration of data</title>
        <p>For each ICEMR project, around 100 different variables
can be used to search and retrieve the data. As indicated in the
Introduction, malaria researchers are interested in mining the
data for insights about the connections between study
participants, their living conditions (dwelling), their health
status (clinical visit), their geographic location and exposure to
mosquitos (entomological measurement data). We assigned the
variables to these five categories based on their mapped
ontology terms taking into account whether they were a
subclass of or having a logical connection to the categories.
With the exception of geographic location, each category had
around 20 different variables that required further grouping to
provide intuitive access to the data for end users. Further
grouping was made based on the ontological understanding of
data. For example, height, weight, and temperature data are all
generated by physical examination. Thus, a new class of data
OGMS_0000083: ‘physical examination’ was added under
category ‘visit’. Using this approach, around 5 different
subtypes were created under each category (except ‘geographic
location’). For example, in addition to ‘physical examination’,
‘medication’, ‘diagnosis’, ‘symptoms’, ‘laboratory findings’,
‘visit type’ and ‘visit details’ were added as subtypes of the
category ‘visit’.</p>
        <p>Term labels used in an ontology are typically chosen for
ontological clarity and can be quite long. As a result, such
labels are often not user-friendly or practical for providing
searches on web sites like PlasmoDB. Alternative display
names were therefore generated for ontology terms. For
example, the display name ‘Age at time of enrollment’ is used
for ontology term EUPATH_0000120: ‘age since birth at time
of enrollment’.</p>
        <p>
          Figure 2 shows the organization of variables that will be
displayed on the website in the three ICEMR projects
discussed here using Protégé, an OWL editor [
          <xref ref-type="bibr" rid="ref17">19</xref>
          ]. Among the
different ICEMR data are found common categories but also
some categories specific to individual projects. Therefore, each
ICEMR project has its own representation of the ICEMR
terminology used as web site search filters to explore its data.
The application of this approach for the Uganda ICEMR
project is shown in Figure 3. The applications for the other
ICEMRs will be very similar and therefore users familiar with
one ICEMR search will also find the other ICEMR searches to
be familiar. Furthermore, the common display and underlying
ontology mappings provide the opportunity for future cross
ICEMR searches.
        </p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>IV. DISCUSSION/ CONCLUSIONS</title>
      <p>Related but different semantic approaches were used to
address the dual challenges of standardizing data dictionaries
across projects and generating user-friendly displays to search
and explore the associated data.</p>
      <p>
        Our approach for standardization is to relate all variables
and associated values to terms from interoperable ontologies
listed at the OBO Foundry. OBO Foundry ontologies provide
the benefit of wide coverage but can also be selectively
imported to create an application ontology such as the EuPath
ontology. When existing terms were not available for mapping,
new ones were created for introduction into the source
ontologies or just placed in the application ontology. The use
of the Basic Formal Ontology (BFO) [
        <xref ref-type="bibr" rid="ref16">20</xref>
        ] by the EuPath
ontology as its upper level greatly facilitated the task of
standardization across projects. BFO models reality rather than
data models and helps interpret when variables and values are
about the same processes, material entities, and measurements.
However, the ontologic semantic organization did not directly
translate well to web site displays for exploring relationships
between study participants, their living conditions, and data
gathered at clinical visits to understand malaria epidemiology.
Instead, categorical organization was better suited for web
display.
      </p>
      <p>An ICEMR terminology was created for the purpose of
web display to organize the standardized variables according to
ways that users are expected to browse them. The ICEMR
terminology also takes into account the need for shortened
names on a web form. Underlying all the terms however is
their basis for understanding through mapping to OBO /
EuPath ontology terms.</p>
      <p>The separation of web display and variable standardization
provides for flexibility in providing different emphases in data
browsing while maintaining the same underlying semantics.
The overall approach has allowed us to achieve the goal of
providing a common system with consistent representation for
the three currently participating ICEMR projects. It also
provides a flexible existing system for introducing data from
other ICEMR projects or other studies of the same type.</p>
    </sec>
    <sec id="sec-6">
      <title>ACKNOWLEDGMENT</title>
      <p>We acknowledge the developers of the Disease Ontology,
the Environmental Ontology and the Drug Ontology for adding
our requested terms into their respective ontologies. G.C.E and
D.G thank Carmen Puemape and Mitchell Guzman for
excellent technical assistance in data management.
[7] The Phenotype And Trait Ontology (PATO) [online]. Available:
https://github.com/pato-ontology/pato/
[8] The Ontology for General Medical Sciences (OGMS) [online].</p>
      <p>Available: https://github.com/OGMS/ogms/
[9] P. L. Buttigieg, N. Morrison, B. Smith, C. J. Mungall, and S. E. Lewis,
“The environment ontology: contextualising biological and biomedical
entities,” J. Biomed. Sem. vol. 4, pp. 43, December 2013.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>J. B.</given-names>
            <surname>Gutierrez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O. S.</given-names>
            <surname>Harb</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Zheng</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. J.</given-names>
            <surname>Tisch</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E. D.</given-names>
            <surname>Charlebois</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. J.</given-names>
            <surname>Stoeckert</surname>
          </string-name>
          , et al.,
          <article-title>“A framework for global collaborative data management for malaria research</article-title>
          ,
          <source>” Am. J. Trop. Med. Hyg</source>
          . vol.
          <volume>93</volume>
          no.
          <issue>3 Suppl.</issue>
          , pp.
          <fpage>124</fpage>
          -
          <lpage>32</lpage>
          ,
          <year>September 2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>C.</given-names>
            <surname>Aurrecoechea</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Brestelli</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. P.</given-names>
            <surname>Brunk</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Dommer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Fischer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Gajria</surname>
          </string-name>
          , et al.,
          <article-title>“PlasmoDB: a functional genomic database for malaria parasites</article-title>
          ,
          <source>” Nucleic Acids Res</source>
          . vol.
          <volume>37</volume>
          , pp.
          <fpage>D539</fpage>
          -
          <lpage>43</lpage>
          ,
          <year>January 2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>V. G.</given-names>
            <surname>Dugan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. J.</given-names>
            <surname>Emrich</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. I.</given-names>
            <surname>Giraldo-Calderón</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O. S.</given-names>
            <surname>Harb</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. M.</given-names>
            <surname>Newman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B. E.</given-names>
            <surname>Pickett</surname>
          </string-name>
          , et al, “
          <article-title>Standardized metadata for human pathogen/vector genomic sequences,"</article-title>
          <source>PloS One</source>
          . vol
          <volume>9</volume>
          no 6, pp.
          <fpage>e99979</fpage>
          ,
          <year>June 2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Ashburner</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Rosse</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Bard</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Bug</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Ceusters</surname>
          </string-name>
          , et al., “
          <article-title>The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration,” Nat Biotechnol</article-title>
          . vol.
          <volume>25</volume>
          , pp.
          <fpage>1251</fpage>
          -
          <lpage>5</lpage>
          ,
          <year>November 2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <surname>A. Musen.. “</surname>
          </string-name>
          <article-title>WebProtégé: a collaborative Web-based platform for editing biomedical ontologies,” Bioinformatics</article-title>
          . vol.
          <volume>30</volume>
          , pp.
          <fpage>2384</fpage>
          -
          <lpage>5</lpage>
          ,
          <year>August 2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>A.</given-names>
            <surname>Bandrowski</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Brinkman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Brochhausen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. H.</given-names>
            <surname>Brush</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Bug</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. C.</given-names>
            <surname>Chibucos</surname>
          </string-name>
          , et al., “
          <article-title>The Ontology for Biomedical Invetigastions,” PLoS One</article-title>
          . vol
          <volume>11</volume>
          no.
          <issue>4</issue>
          , pp.
          <fpage>e0154556</fpage>
          ,
          <year>April 2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>W.A.</given-names>
            <surname>KIbbe</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Arze</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Felix</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Mitraka</surname>
          </string-name>
          , E. Bolton,
          <string-name>
            <given-names>G.</given-names>
            <surname>Fu</surname>
          </string-name>
          , et al.,
          <source>“Disease Ontology</source>
          <year>2015</year>
          update
          <article-title>: an expanded and updated database of human diseases for linking biomedical knowledge through disease data</article-title>
          ,
          <source>” Nucleic Acids Res</source>
          . vol.
          <volume>43</volume>
          , pp.
          <fpage>D1071</fpage>
          -
          <lpage>8</lpage>
          ,
          <year>January 2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>J.</given-names>
            <surname>Hanna</surname>
          </string-name>
          , E. Joseph,
          <string-name>
            <given-names>M.</given-names>
            <surname>Brochhausen</surname>
          </string-name>
          , and
          <string-name>
            <given-names>W. R.</given-names>
            <surname>Hogan</surname>
          </string-name>
          , “
          <article-title>Building a drug ontology based on RxNorm and other sources,”</article-title>
          <string-name>
            <given-names>J.</given-names>
            <surname>Biomed</surname>
          </string-name>
          . Sem. vol.
          <volume>4</volume>
          , pp.
          <fpage>44</fpage>
          ,
          <year>December 2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>L. G.</given-names>
            <surname>Cowell</surname>
          </string-name>
          and
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , “
          <article-title>Infectious disease ontology,” in Infectious disease informatics</article-title>
          , Springer New York,
          <year>2010</year>
          , pp.
          <fpage>373</fpage>
          -
          <lpage>395</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>P. N.</given-names>
            <surname>Robinson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Köhler</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Bauer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Seelow</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Horn</surname>
          </string-name>
          , and
          <string-name>
            <given-names>S.</given-names>
            <surname>Mundlos</surname>
          </string-name>
          , “
          <article-title>The Human Phenotype Ontology: a tool for annotating and analyzing human hereditary disease</article-title>
          ,
          <source>” Am. J. Hum. Genet</source>
          . vol.
          <volume>83</volume>
          , pp.
          <fpage>610</fpage>
          -
          <lpage>5</lpage>
          ,
          <year>November 2008</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [14]
          <article-title>The Information Artifact Ontology (IAO) [Online]</article-title>
          . Available: https://github.com/information-artifact-ontology/IAO/
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>M.</given-names>
            <surname>Brochhausen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Zheng</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Birtwell</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Williams</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. M.</given-names>
            <surname>Masci</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H. J.</given-names>
            <surname>Ellis</surname>
          </string-name>
          , et al.,
          <article-title>“OBIB - a novel ontology for biobanking,”</article-title>
          <string-name>
            <given-names>J.</given-names>
            <surname>Biomed</surname>
          </string-name>
          . Sem. vol.
          <volume>7</volume>
          , pp.
          <fpage>23</fpage>
          , May 2016
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [16]
          <article-title>The Symptom Ontology (SYMP) [Online]</article-title>
          . Available: http://symptomontologywiki.igs.umaryland.edu/mediawiki/index.php
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>C.</given-names>
            <surname>Jonquet</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. H.</given-names>
            <surname>Shah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. A.</given-names>
            <surname>Musen</surname>
          </string-name>
          , “
          <article-title>The open biomedical annotator” Summit on Translat Bioinforma</article-title>
          . vol.
          <year>2009</year>
          , pp.
          <fpage>56</fpage>
          -
          <lpage>60</lpage>
          ,
          <year>March 2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>P. L.</given-names>
            <surname>Whetzel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. F.</given-names>
            <surname>Noy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. H.</given-names>
            <surname>Shah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. R.</given-names>
            <surname>Alexander</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Nyulas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Tudorache</surname>
          </string-name>
          , et al.,
          <article-title>“BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications</article-title>
          ,
          <source>” Nucleic Acids Res</source>
          . vol.
          <volume>39</volume>
          , pp.
          <fpage>W541</fpage>
          -
          <lpage>5</lpage>
          ,
          <year>July 2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>R.</given-names>
            <surname>Arp</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Smith</surname>
          </string-name>
          , and
          <string-name>
            <given-names>A. D.</given-names>
            <surname>Spear</surname>
          </string-name>
          , “
          <article-title>Building ontologies with Basic Formal Ontology</article-title>
          ,” The MIT Press,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [19]
          <string-name>
            <surname>Protégé</surname>
          </string-name>
          [Online]: Available: http://protege.stanford.edu
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>