<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Standardized Drug and Pharmacological Class Network Construction</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Qian Zhu</string-name>
          <email>zhu.qian@mayo.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Guoqian Jiang</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Liwei Wang</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Christopher G. Chute</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Health Sciences Research, Division of Biomedical Statistics and Informatics, Mayo Clinic</institution>
          ,
          <addr-line>Rochester, MN</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>School of Public Health, Jilin University</institution>
          ,
          <addr-line>Changchun, Jilin</addr-line>
          ,
          <country country="CN">China</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Dozens of drug terminologies and resources capture the drug and/or drug class information, ranging from their coverage and adequacy of representation. No transformative ways are available to link them together in a standard way, which hinders data integration and data representation for drug-related clinical and translational studies. In this paper, we introduce our preliminary work for building a standardized drug and drug class network that integrates multiple drug terminological resources, using Anatomical Therapeutic Chemical (ATC) and National Drug File Reference Terminology (NDF-RT) as network backbone, and expanding with RxNorm and Structured Product Label (SPL). In total, the network consists of 39,728 drugs and drug classes. Meanwhile, we calculated and compared structure similarity for each drug / drug class pair from ATC and NDF-RT, and analysed constructed drug class network from chemical structure perspective.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>INTRODUCTION</title>
      <p>
        Drug classes are group names for drugs that have similar
activities or are used for a same type of disease and
disorder. There are different ways to classify drugs. One way is
to group drugs based on their therapeutic use or class (e.g.,
antiarrhythmic or diuretic drugs) as used by Anatomical
Therapeutic Chemical (ATC) [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Another way is to group
drugs using their dominant mechanism of action as used by
National Drug File Reference Terminology (NDF-RT) [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ].
However, drug classes defined by different systems are not
compatible. It is worth to compare and integrate them in a
universal fashion in order to support clinical related studies
better. For example, Mougin, et al. [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] conducted a study for
comparing drug classes between ATC and NDF-RT
focusing on the relations between drugs and pharmacological
classes (i.e., drug-class membership relations), which will
facilitate the integration of these two resources.
      </p>
      <p>Drug terminologies define drug entities as well as relevant
properties and relationships with pharmacological classes.
Drug terminologies are usually developed and maintained
by different institutions using site-specific drug coding
systems. Heterogeneous drug representations across different
systems make it difficult to navigate diverse drug resources.
The lack of a transformative way to link heterogeneous drug
resources hinders data integration and data representation
for drug-related clinical and translational studies. To
overcome this obstacle, we proposed to represent drug
information from diverse resources in a standard and integrated
manner.</p>
      <p>
        ATC and NDF-RT are the proposed sources of drug
classification information. In the present study, we developed an
approach to map drug and drug class entities from ATC and
NDF-RT to UMLS (Unified Medical Language System) [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]
and generated these mappings as a drug network backbone.
Furthermore, we extended such network with RxNorm [5]
and Structured Product Labeling (SPL) [
        <xref ref-type="bibr" rid="ref5">6</xref>
        ] integration,
benefited from the broad drug relevant knowledge provided by
these two resources. RxNorm provides links among
different vocabularies, e.g. NDF-RT. SPL contains full drug
interaction information, such as drug and drug interaction, and
adverse drug event, etc., which has been explored and
implemented by investigators and relevant applications have
been developed, such as LinkedSPLs [
        <xref ref-type="bibr" rid="ref6">7</xref>
        ], ADEPedia [
        <xref ref-type="bibr" rid="ref7">8</xref>
        ].
Additionally, to extend and compare the drug classes
defined by ATC and NDF-RT from chemical structure point
of view, we introduced chemical structure similarity with
the assumption that similar molecules have similar
activities.
      </p>
      <p>The paper is organized in several sections. We introduce the
background knowledge about the resources and tools used
in material section; in the methods section, we introduce the
workflow details for network construction; then followed by
discussion and conclusion sections.
2</p>
    </sec>
    <sec id="sec-2">
      <title>MATERIALS AND METHODS</title>
      <p>
        NDF-RT is a well-known drug terminological resource, and
snapshot of NDF-RT was downloaded as of Nov. 8, 2012.
In ATC classification system, drugs are categorized into
different groups at five different levels according to the
organ or system on which they act and/or their therapeutic and
chemical characteristics [
        <xref ref-type="bibr" rid="ref8">9</xref>
        ]. ATC with a released version on
January 2012 was used in this study. RxNorm provides
normalized names for clinical drugs and links them to
several drug vocabularies differentiating by “SAB” label. For
example, “SAB=MTHSPL” indicates the source from SPL
and “SAB=NDFRT” from NDF-RT. Two files are used in
this study: 1) RXNCONSO.RRF, including all connections
with source vocabularies. 2) RXNREL.RRF including
relationships among concepts. RxNorm used in this study was
the version of Oct. 2012. SPL contains structured content of
labeling (all text, tables and figures), along with additional
machine readable information. The mappings between SPL
and RxNorm used in this study are extracted from RxNorm
RXNCONSO files with SAB = MTHSPL.
      </p>
      <p>In this paper, we introduce a drug and drug class network by
utilizing multiple drug terminological resources: ATC,
NDF-RT, RxNorm, and SPL. ATC and NDF-RT are used as
the network backbone, from which we integrated RxNorm
and SPL as extension. Meanwhile, we calculated structure
similarity for drug pairs from ATC and NDF-RT, and
clustered them by structural similarity. The details of each step
conducted in this study are described in the following
sections.
2.1</p>
      <sec id="sec-2-1">
        <title>Mapping NDF-RT with ATC</title>
        <p>To map NDF-RT with ATC via UMLS, we translated NUI,
NDF-RT Numerical Unique Identifier, and ATC name to
UMLS CUI, UMLS concept unique identifier.</p>
        <sec id="sec-2-1-1">
          <title>3.1.1 ATC mapping to UMLS</title>
          <p>
            ATC is not well integrated with other drug terminologies
(e.g., NDF-RT), as it uses its own coding system to code
drug entities. To map ATC with NDF-RT and present the
drug network transformatively by using standard
representation, UMLS, we employed NCBO annotator [
            <xref ref-type="bibr" rid="ref9">10</xref>
            ] to
semantically annotate each ATC name. Among more than 200
ontologies from UMLS Metathesaurus and NCBO BioPortal
[
            <xref ref-type="bibr" rid="ref10">11</xref>
            ], RxNorm and NDF-RT have higher priority in this
study. To avoid unnecessary annotations by non-drug
relevant ontologies, we limited UMLS semantic types [
            <xref ref-type="bibr" rid="ref11">12</xref>
            ]
within “Chemicals &amp; Drugs” semantic group [
            <xref ref-type="bibr" rid="ref12">13</xref>
            ]. We
extracted ontology id and concept id, which are two
mandatory input parameters to invoke NCBO BioPortal REST API
[
            <xref ref-type="bibr" rid="ref13">14</xref>
            ] for searching UMLS CUI, from the annotation results.
          </p>
        </sec>
        <sec id="sec-2-1-2">
          <title>3.1.2 NDF-RT mapping to RxNorm and UMLS</title>
          <p>NDF-RT concepts are organized into different categories
with corresponding category labels. For example,
“N0000179008, 1,1,1-trichloroethane,
[Chemical/Ingredient]” and “N0000175641, Autonomic Ganglionic
Blocker, [EPC]” are chemical ingredient and EPC class
respectively. In this study, we retrieved the concepts that are
labeled as VA class, VA product, EPC, Chemical ingredient
and generic ingredient combination.</p>
          <p>
            SQL query was executed to search RxCUIs (RxNorm
Concept Unique Identifier) from RxNorm RXNCONSO table
that was pre-loaded into our local MySQL database for
NUIs. We retrieved UMLS CUI by invoking NLM RxNav
RESTful API [
            <xref ref-type="bibr" rid="ref14">15</xref>
            ] with each NUI as an input parameter.
2.2
          </p>
        </sec>
      </sec>
      <sec id="sec-2-2">
        <title>Calculating structure similarity</title>
        <p>
          To analyze and expand the drug and drug class network
from chemical structure perspective, we calculated the
structure similarity among the drug pairs from ATC and
NDF-RT, and grouped them using the score of structure
similarity as Tanimoto Coefficient, i.e., similarity between
these pairs of descriptors [
          <xref ref-type="bibr" rid="ref15">16</xref>
          ]. The cutoff value of the
structure similarity is set as the score greater than 0.85, as it
exhibits similar biological activity between the two molecules.
We first converted NDF-RT drug name and ATC name to
SIMILES (Simplified molecular-input line-entry system)
[
          <xref ref-type="bibr" rid="ref16">17</xref>
          ] as chemical representation by invoking PubChem
entrez web service [
          <xref ref-type="bibr" rid="ref17">18</xref>
          ] and NCI resolver [
          <xref ref-type="bibr" rid="ref18">19</xref>
          ] REST API.
Then we translated SMILES to chemical fingerprint and
calculated Tanimoto similarity by using the aforementioned
CDK functions.
2.3
        </p>
      </sec>
      <sec id="sec-2-3">
        <title>Integrating RxNorm and SPL mappings</title>
        <p>Mappings among RxNorm, SPL and NDF-RT are provided
by RxNorm and available in the RxNorm RXNCONSO
table. Two steps were performed to retrieve these mappings.
First, we obtained concepts labeled as “SAB=NDFRT” and
“SAB=RXNORM”, denoted as RxNorm and NDF-RT
mappings. Then, we searched for the concepts with
“SAB=MTHSPL” label from the concepts identified in the
first step. Then the final list of concepts is the common
concepts across the three resources.</p>
        <p>The network has been expanded from NDF-RT nodes that
have mappings with RxNorm and SPL. We extracted SPL
identifier (setId) from RXNREL table and saved for future
SPL relevant information, LinkedSPL integration.
In addition, we performed a case study to demonstrate the
usefulness of the drug and drug class network.
3</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>RESULTS</title>
      <p>There are total 5,717 individual entities, which correspond
to 4,483 distinct ATC names, i.e. one drug can be
categorized into multiple therapeutic classes (more details
described in the Discussion section).</p>
      <p>Of 48,266 NDF-RT concepts, 34,011 concepts were used in
this study, consisting of 15,857 VA Products, 486 VA
classes, 9,960 Chemical/Ingredients, 7,184 Generic Ingredient
Combinations, and 524 EPC. The child and parent
relationships among these NDF-RT concepts are retrieved and
stored from RxNorm RXNREL table via “CHD” (concept 1
is a child of concept 2) and “PAR” (concept 1 is a parent of
concept 2) labels.</p>
      <p>RxNorm, SPL and NDF-RT mappings were extracted from
two RxNorm files: RXNCONSO and RXNREL, which
were loaded into MySQL database.
3.1</p>
      <sec id="sec-3-1">
        <title>Results for ATC and NDF-RT mappings</title>
        <p>In order to build drug and drug class network with ATC and
NDF-RT as backbone, first of all, we mapped ATC entities
with NDF-RT concepts via UMLS, four steps involved.</p>
        <sec id="sec-3-1-1">
          <title>4.1.1 ATC Annotated by NCBO</title>
          <p>
            Standardized Drug and Pharmacological Class Network Construction
3,607 ATC entities including 3,152 drugs and 455 drug
classes were mapped to UMLS CUIs by two ontologies,
RxNorm and NDF-RT from NCBO BioPortal. Of these
3607 ATC mappings, 2180 ATC entities were exactly
matched with the preferred names from RxNorm and
NDFRT. 866 ATC entities including 211 drug classes and 655
drugs were mapped to other ontologies available from
NCBO. There are 1,244 ATC entities (21.8%) including 657
drugs and 587 drug classes failed to map to UMLS due to
no annotations generated accordingly. We attempted to map
these failed ATC names with RxNorm directly by invoking
NLM RxNav RESTful API [
            <xref ref-type="bibr" rid="ref19">20</xref>
            ] with ATC names as input
parameter, but none of them got mapping results. The
failure reasons are discussed in the discussion section further.
          </p>
        </sec>
        <sec id="sec-3-1-2">
          <title>4.1.2 NCBO annotation evaluation</title>
          <p>The annotations were automated programmatically using
NCBO Annotator Web Services API. We manually
evaluated the annotation results. Of the 4,473 annotations with
NDF-RT and RxNorm, 2,401 exact mappings were not
further evaluated. The authors (QZ, LW) manually reviewed
the rest of annotations (2,072 in total). As the evaluation
results, 88.7% is correct, 10.3% is partial mappings, and
1.0% is incorrect. The precision was calculated as 99.5%,
recall as 78.2% and F-measure as 87.4%, in which we
counted exact mappings, partial mappings and correct
mappings (4,453 in total) as true positive, 1,244 failed mappings
as false negative and 20 incorrect mappings as false
positive.</p>
        </sec>
        <sec id="sec-3-1-3">
          <title>4.1.3 Mapping NDF-RT to RxNorm and UMLS</title>
          <p>NDF-RT and RxNorm mappings exist in the RXNCONSO
table with “SAB=NDFRT” label. Consequently, RxCUI
corresponding to each NDF-RT concept can be retrieved
from these mappings directly.</p>
          <p>
            NDF-RT provides UMLS mappings. Hence, to retrieve
UMLS for each NDF-RT concept, we called NLM NDF-RT
RESTful API [
            <xref ref-type="bibr" rid="ref8">9</xref>
            ]. The searching results are shown in Table
1. 99.2% NDF-RT concepts have been mapped to UMLS.
          </p>
          <p>NDF-RT Concepts
Chemical/Ingredient (9,960)</p>
          <p>VA Class (486)
VA Product (15,857)</p>
          <p>EPC (524)
Generic ingredient combination</p>
          <p>(7,184)
Total (34,011)</p>
          <p>NUI
9,934
486
15,695
480
7,139
33,734</p>
        </sec>
        <sec id="sec-3-1-4">
          <title>4.1.4 ATC and NDF-RT mapping</title>
          <p>In total, 3,850 distinct mappings between ATC and
NDFRT were generated, including 2,015 chemical/ingredients,
1,826 Generic Ingredient Combinations and 1 VA class. It
includes distinct 2,226 ATC entities, covering 99 drug
classes, and 2,127 individual drugs.</p>
        </sec>
      </sec>
      <sec id="sec-3-2">
        <title>Results for structural similarity calculation</title>
        <p>SMILES have been retrieved for all drugs from ATC and
NDF-RT via PubChem Entrez web API and NCI Resolver
web API. 2,618 ATC entities have gotten SMILES from
NCI, 3,471 entries retrieved from PubChem. Combining
NCI and PubChem searching results, total 3,487 ATC
entries got SMILES, and 9,132 unique NDF-RT concepts got
SMILES.</p>
        <p>We calculated the Tanimoto coefficient as structure
similarity for each pair of concepts from ATC and NDF-RT
separately by converting SMILES to fingerprint. Then we got
8,513 pairs from ATC and 69,882 pairs from NDF-RT with
Tanimoto coefficient greater than 0.85, and integrated them
into the drug and drug class network.
3.3</p>
      </sec>
      <sec id="sec-3-3">
        <title>Results for NDF-RT, RxNorm and SPL mapping</title>
        <p>We integrated RxNorm and SPL mappings with NDF-RT.
The mappings between RXNORM, NDF-RT and SPL
resulted in 5,838 unique RxNorm concepts with 36,408
NDFRT concepts and 41,188 SPL labels. The mappings mostly
fall into two main categories according to term types
defined by RxNorm, 3,056 are Semantic Clinical Drugs and
1,543 are Ingredients.</p>
        <p>It is worthy to note that one RxNorm concept may be
mapped to multiple NDF-RT and/or SPL concepts, for
example, RxCUI “74” mapped to 3 NUIs in NDF-RT
including N0000006481, N0000147349, N0000006481 and 11
set_ids in MTHSPL such as
0d65128b-8eb7-440b-870a7e3be18152b3,1e6d6cd5-ab14-4258-a0fe-5f6a3cae437f.
4</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>DISCUSSION</title>
      <p>In this study, we successfully built a drug and drug class
network with 39,728 concepts from ATC and NDF-RT. All
concepts were mapped to UMLS and labeled as UMLS
CUIs accordingly. We also integrated RxNorm and SPL
mappings, and extended the network with structure
similarity calculation.
4.1</p>
      <sec id="sec-4-1">
        <title>ATC to UMLS mapping</title>
        <p>
          In total, 77.9% ATC terms have been mapped to UMLS.
Comparing to 68.7% mapping results conducted by Merabti
et al [
          <xref ref-type="bibr" rid="ref20">21</xref>
          ], our study shows the improvement of mappings
from ATC to UMLS by leveraging NCBO annotator.
However, 22.1% ATC terms failed to be mapped due to several
reasons as follows, 1) Many of the ATC terms are
combinations of multiple concepts, such as “calcium acetate and
magnesium carbonate”, “combinations of sulfonamides and
trimethoprim, including derivatives”; 2) The exclusions are
embedded in the ATC names, such as “platelet aggregation
inhibitors excluding heparin”, “nutrients without
phenylalanine; 3) Non-standard representation is used by ATC though
we corrected and expanded some abbreviations occurring in
        </p>
        <p>
          ATC name. For example, “DIGESTIVES, INCL.
ENZYMES” was corrected to “DIGESTIVES,
INCLUDING ENZYMES”; 4) Non-drug terms are used,
especially for drug classes in ATC, such as “VARIOUS”,
“SENSORY ORGANS”. Above obstacles were the main
reasons for mapping failure. In the future study, we will
explore MMTx program that reported by Mougin et al. [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ],
and more NLP (Nature Language Processing) algorithms to
parse ATC names for improving the mapping performance
between the ATC and the UMLS.
4.2
        </p>
      </sec>
      <sec id="sec-4-2">
        <title>Benefits from structure similarity integration</title>
        <p>Structure similarity calculation applied in this study enables
connections among the drug nodes sharing common similar
chemical substructures. Beside the benefit shown in the case
study, this integration also provides relevant clues for
guiding clinical decision support system from the structure
perspective as it offers a full profile of therapeutics for
individual drugs. ATC classification system categorizes drugs
according to its therapeutic classes; hence, one ATC drug can
be grouped into multiple categories due to its diverse
therapeutic functionalities. For instance, “Thonzylamine” is an
antihistamine and anticholinergic used as an antipruritic and
is grouped into two categories: “antiallergic agents” and
“antihistamines for topical use” within the ATC hierarchy.
The corresponding two ATC entities (R01AC06 and
D04AA01) for “Thonzylamine” in two separate classes (“R”
and “D”) are connected based on similarity score that is
equal to 1. Thus, the entities within these two categories are
connected, and physicians would be able to utilize such
knowledge for Thonzylamine for their clinical decision
from both therapeutics and structure point of view.
4.3</p>
      </sec>
      <sec id="sec-4-3">
        <title>Future work</title>
        <p>Drug entity mapping algorithm will be modified to enable
more connections detected; more human review will be
expected to improve the accuracy of the mappings.
Meanwhile, we will seek possible collaborations with external
sites such as the NLM for improving such mapping
algorithm development. We will integrate more drug related
resources, such as Drugbank and PharmGKB, and drug
interaction data, drug and adverse event data as shown in
Figure 1. The entire data set generated in this project will be
released to public once the proposed action items
accomplished.
5</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>CONCLUSION</title>
      <p>We successfully integrated NDF-RT, ATC, RxNorm and
SPL and built a drug and drug class network using
standardized identifier for representing drug and drug class entities.
In addition, the network was expanded from chemical
structure perspective by similarity calculation. More other drug
terminological resources and drug interaction information
will be integrated in the future study.</p>
    </sec>
    <sec id="sec-6">
      <title>ACKNOWLEDGMENTS</title>
      <p>This work was supported by the Pharmacogenomic
Research Network (NIH/NIGMS-U19 GM61388) and the
SHARP Area 4: Secondary Use of EHR Data
(90TR000201).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [ 1 ] ATC: http://www.who.int/classifications/atcddd/en/.
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <surname>NDF-RT</surname>
          </string-name>
          : http://www.nlm.nih.gov/research/umls/sourcereleasedocs/current/NDFRT/.
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <surname>Mougin</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Burgun</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Bodenreider</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          <article-title>Comparing Drug-Class Membership in ATC and NDF-RT</article-title>
          .
          <source>Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium</source>
          ,
          <year>2012</year>
          :
          <fpage>437</fpage>
          -
          <lpage>443</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Bodenreider</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          <article-title>The Unified Medical Language System (UMLS): integrating biomedical terminology</article-title>
          .
          <source>Nucleic Acids Res</source>
          .
          <year>2004</year>
          ,
          <volume>32</volume>
          ,
          <fpage>267</fpage>
          -
          <lpage>270</lpage>
          [ 5 ] RxNorm: www.nlm.nih.gov/research/umls/rxnorm. Accessed by Apr.
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>Structured</given-names>
            <surname>Product</surname>
          </string-name>
          Labeling: http://www.fda.gov/ForIndustry/DataStandards/StructuredProductLabeling/ default.htm.
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [7]
          <string-name>
            <surname>Hassanzadeh</surname>
            <given-names>O</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhu</surname>
            <given-names>Q</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Freimuth</surname>
            <given-names>R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Boyce</surname>
            <given-names>R</given-names>
          </string-name>
          ,
          <article-title>Extending the "Web of Drug Identity" with Knowledge Extracted from United States Product Labels</article-title>
          , submitted to AMIA
          <source>Summit on Clinical Research Informatics</source>
          ,
          <year>2013</year>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Jiang</surname>
            <given-names>G</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Solbrig</surname>
            <given-names>H. R</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chute</surname>
            <given-names>C.G.</given-names>
          </string-name>
          <article-title>ADEpedia: a scalable and standardized knowledge base of Adverse Drug Events using semantic web technology</article-title>
          .
          <source>AMIA Annu Symp Proc</source>
          .
          <year>2011</year>
          :
          <fpage>607</fpage>
          -
          <lpage>16</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [ 9 ]http://en.wikipedia.org/wiki/Anatomical_Therapeutic_Chemical_
          <article-title>Classif ication_System. Accessed by Apr</article-title>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [10]
          <string-name>
            <surname>Jonquet</surname>
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shah</surname>
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Musen</surname>
            <given-names>M.</given-names>
          </string-name>
          <article-title>The Open Biomedical Annotator</article-title>
          .
          <source>AMIA Summit on Translational Bioinformatics;</source>
          <year>2009</year>
          :
          <fpage>56</fpage>
          -
          <lpage>60</lpage>
          . The NCBO Annotator web service: http://www.bioontology.org/annotator-service.
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [11]
          <string-name>
            <surname>Noy</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Shah</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dai</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dorf</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , Gri_th, N.,
          <string-name>
            <surname>Jonquet</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Montegut</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rubin</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Youn</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Musen</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Bioportal: A web repository for biomedical ontologies and data resources</article-title>
          .
          <source>In: Demo session at 7th International Semantic Web Conference (ISWC</source>
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>Semantic</given-names>
            <surname>Type</surname>
          </string-name>
          : http://www.nlm.nih.gov/research/umls/META3_
          <article-title>current_semantic_types.ht ml</article-title>
          .
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .2013
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Bodenreider</surname>
            <given-names>O</given-names>
          </string-name>
          ,
          <string-name>
            <surname>McCray</surname>
            <given-names>AT</given-names>
          </string-name>
          <article-title>Exploring semantic groups through visual approaches</article-title>
          .
          <source>Journal of Biomedical Informatics</source>
          <year>2003</year>
          ;
          <volume>36</volume>
          (
          <issue>6</issue>
          ):
          <fpage>414</fpage>
          -
          <lpage>432</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [14]
          <string-name>
            <surname>BioPortal</surname>
            <given-names>REST</given-names>
          </string-name>
          services: http://www.bioontology.org/wiki/index.php/NCBO_REST_services.
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [15]
          <string-name>
            <surname>NDF-RT RESTful</surname>
            <given-names>API</given-names>
          </string-name>
          : http://rxnav.nlm.nih.gov/NdfrtRestAPI.html#label:r24.
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Holliday</surname>
            <given-names>JD</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hu</surname>
            <given-names>CY</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Willett</surname>
            <given-names>P</given-names>
          </string-name>
          ,
          <article-title>Grouping of coefficients for the calculation of inter-molecular similarity and dissimilarity using 2D fragment bitstrings</article-title>
          .
          <source>Comb Chem High Throughput Screen</source>
          ,
          <year>2002</year>
          ,
          <volume>5</volume>
          (
          <issue>2</issue>
          ):
          <fpage>155</fpage>
          -
          <lpage>66</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [17] SMILES: http://en.wikipedia.org/wiki/Simplified_molecularinput_
          <article-title>line-entry_system</article-title>
          .
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [18] PubChem Entrez: http://www.ncbi.nlm.nih.gov/books/NBK25500/. Accessed by Apr.
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [19 ] NCI resolver: http://cactus.nci.nih.gov/chemical/structure. Accessed by Apr.
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [20]
          <string-name>
            <surname>RxNorm RESTful</surname>
            <given-names>API</given-names>
          </string-name>
          : http://rxnav.nlm.nih.gov/RxNormRestAPI.html.
          <source>Accessed by Apr</source>
          .
          <volume>11</volume>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [21]
          <string-name>
            <surname>Merabti</surname>
          </string-name>
          et al,
          <year>2011</year>
          , Stud Health Technol Inform.
          <year>2011</year>
          ;
          <volume>166</volume>
          :
          <fpage>206</fpage>
          -
          <lpage>13</lpage>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>