To MIREOT or not to MIREOT? A case study of the impact of using MIREOT in the Experimental Factor Ontology (EFO)

Abstract

MIREOT is a mechanism for the selective re-use of individual ontology classes in other ontologies. Designed to minimise effort and to support orthogonality, it is now in widespread use. The consequences for ontology integrity and automated reasoning of using the MIREOT mechanism have so far not been fully assessed. In this paper, we perform an analysis of the Experimental Factor Ontology (EFO), an ontology which uses the MIREOT process to gather classes from a large range of other ontologies. Our study examines the effect of combining EFO with the ontologies it references by actually importing them into the EFO. We then evaluate the consistency and status of the combined ontologies. Through our investigation, we reveal that EFO in combination with all its referenced ontologies is logically inconsistent. Furthermore, when EFO is individually combined with many of the ontologies it references, we find a large number of unsatisfiable classes. These results demonstrate a potential problem within a major ontological ecosystem, and reveals possible disadvantages to the use of the MIREOT system for developing ontologies.

INTRODUCTION

There has recently been some discussion regarding best practice in developing ontologies so that they become useful artifacts for a particular purpose (Hoehndorf et al., 2013) and some applicationoriented recommendations have been made, which aim to make ontologies more usable in a wider range of application domains. In particular, work with ontologies can easily become complex and very costly in terms of computational time and space (i.e., memory), and so guidelines have been developed to make ontologies more readily usable with the current state of technology available to researchers (Courtot et al., 2011) .

Many ontologies include and reference classes from other ontologies. The ability to reuse existing classes is one of the most attractive features of ontologies, to the extent that it is even required for ontologies that participate in the OBO Foundry initiative (Smith et al., 2007) . Recently several ontologies have been developed for specific applications which primarily consist of classes that are imported from other ontologies, and are combined in a new way suitable for the intended application. The Web Ontology Language (OWL) (Grau et al., 2008) contains a method to include (i.e., import) a complete ontology, identified by its IRI, into another ontology so that all the axioms of the imported ontology become available within the importing ontology. This method essentially creates an ontology in which the axioms from multiple ontologies (or multiple ontology files) are merged, and thereby provides a simple form of modularity. However, the practical problem with this approach is that, depending on the size of the imported ontologies, it can sometimes render working with the combined ontology impossible given limited hardware resources.

Many ontologies which contain classes that are widely reused, such as the NCBI Taxonomy (Sayers et al., 2009) or the Gene Ontology (GO) (Ashburner et al., 2000) , are very large, and importing them into another ontology, editing them on their own with tools such as Protege (Noy et al., 2001) , or using them in conjunction with OWL reasoners such as HermiT (Motik et al., 2009) is prohibitively expensive and time-consuming, if at all possible. A popular solution to this problem, particularly in the biomedical and biological domains, is the MIREOT (Minimum Information to Reference an External Ontology Term) method (Courtot et al., 2011) , which is a set of guidelines first published in 2009 for importing classes from an external ontology when building an ontology. Essentially, MIREOT provides a way for ontology creators to reference individual classes from external ontologies without actually importing the full ontology. Instead, the IRI of the class from the referenced ontology is used, while all axioms from the source ontology are omitted (except a single subclass axiom to the direct superclass of the referenced class). MIREOT also allows the import of axioms from the referenced ontology, and inclusion of parent classes recursively – however, this option is not frequently used. For example, to use the class cell (http:// purl.obolibrary.org/obo/GO_0005623) in an ontology through the MIREOT method, it would suffice to use the class IRI from GO (i.e., http://purl.obolibrary.org/obo/ GO_0005623), add the axiom that cell is a subclass of cellular component, and ignore all other axioms from GO.

Using the MIREOT method avoids the overheads involved in importing complete ontologies. Additionally, the explicit aim of the method is also to prevent inconsistency and unintended inferences by encouraging the reuse of classes that are already well-defined and established within the domain (Courtot et al., 2011) . Consequently, many ontologies make use of the MIREOT method. One of these ontologies is the Experimental Factor Ontology (EFO) (Malone et al., 2010) . The EFO is an ontology initially developed in the context of the ArrayExpress and Gene Expression Atlas databases (Parkinson et al., 2011; Kapushesky et al., 2010) to annotate data from gene expression experiments. It is now also applied outside this domain for the annotation of CTTV database content and mapping disease and phenotype data from the literature (Sarntivijai et al., 2016) . EFO contains classes from ontologies in many domains, including anatomy (e.g., UBERON (Mungall et al., 2012) ), chemistry (ChEBI (Degtyarenko et al., 2007) ), and GO (Ashburner et al., 2000) . As of April 2016, it has a total of 17,947 classes.

Here, we provide a case-study on the consequences of applying the MIREOT method to the EFO. We evaluate the consistency and coherency of EFO, and the resulting effects on semantic interoperability. We find that the use of MIREOT in EFO has led to the accumulation of a significant amount of hidden logical contradictions. 2 2.1

MATERIALS AND METHODS

Ontologies and ontology versions

For our experiments, we use the EFO, downloaded from http:// aber-owl.net/ontology/EFO/download on 2016-04-22.

EFO references classes from a large number of ontologies. Given the set of class IRIs in EFO, we developed an approach for identifying which ontology the “defining” ontology is, i.e., which ontology holds the authoritative set of axioms and definitions for the class. Although the MIREOT guidelines state that ontologies shall include the IRI of the source ontology when referencing an external ontology class (Courtot et al., 2011) , we found that this was not the case in the EFO. To identify which ontology is the reference ontology within which the class was originally defined, we lexically matched the list of class IRIs against a list of ontology IDs (e.g. CL for the Cell Ontology) acquired from the AberOWL repository (Hoehndorf et al., 2015) . We then manually curated these results and found that EFO does not in fact reference FO or ATO, and that these were matched simply because they were substrings of other ontology IDs. We removed these from our set of ontologies manually. Additionally, we discovered that the combination of IDO and EFO produces an unloadable ontology, i.e., a syntax error. Therefore, we also excluded the IDO from the set of ontologies in our experiments. Table 1 shows all the ontologies from which EFO references classes. We obtained all the referenced ontologies on 2016-04-22. 2.2

Implementation and experimental setup

For all experiments, we use the OWLAPI 4.1.0 (Horridge et al., 2007) . To classify the ontologies, we use the Elk reasoner version 0.4.2 (Kazakov et al., 2014) . Elk supports the OWL 2 EL profile, a fragment of OWL that supports tractable (i.e., polynomial-time) reasoning, but which lacks support for many logic operators. In particular, OWL 2 EL does not support the use of negation in class descriptions or use of the universal quantifier. The only type of axiom in OWL 2 EL that could result in an explicit contradiction (i.e., the inference of an unsatisfiable class or the detection of an inconsistency in the ontology, usually through instantiation of an unsatisfiable class) is the disjointness axiom. ID ERO UBERON CL ORDO CHEBI BTO TO GO HP PATO EO PO OBI DOID SO IAO MP MPATH FBbt ZEA PR IDO OGMS

Name Eagle-i Resource Ontology Uber Anatomy Ontology Cell Ontology Orphanet Rare Disease Ontology Chemical Entities of Biological Interest Brenda Tissue Ontology Plant Trait Ontology Gene Ontology Human Phenotype Ontology Phenotypic Quality Ontology Plant Environment Ontology Plant Ontology Ontology for Biomedical Investigations Human Disease Ontology Sequence types and features Ontology Information Artifact Ontology Mammalian Phenotype Ontology Mouse Pathology Ontology Drosopihilia Anatomy Ontology Maize Gross Anatomy Ontology Protein Domains Ontology Infectious Disease Ontology

Ontology for General Medical Science

We further used the FaCT++ reasoner (Tsarkov and Horrocks, 2006) to classify EFO with its imports, in order to identify any cases in which additional unsatisfiability would be revealed through the violation of more expressive axioms. 3 3.1

RESULTS AND DISCUSSION

Evaluating the consistency of EFO with respect to referenced ontologies

For each ontology referenced in EFO, we created a new ontology consisting of EFO and adding an explicit import statement to the referenced ontology. We then use an OWL 2 EL reasoner and test the consistency of this ontology. An ontology is consistent if it contains no contradictions. If the ontology is consistent, we record the number of unsatisfiable classes. A class is unsatisfiable if it cannot have any instances.

Table 2 provides an overview of the ontologies that, when imported into EFO, result in unsatisfiable classes. For example, importing the UBERON ontology (Mungall et al., 2012) results in a large number (650) of unsatisfiable classes, and, similarly, importing the Mammalian Phenotype Ontology (MP) (Smith et al., 2004) results in 728 unsatisfiable classes.

To determine the effect of axioms in more expressive logics, i.e., OWL 2 DL, we further used the FaCT++ reasoner (Tsarkov and Horrocks, 2006) and attempted to classify each ontology, i.e., EFO together with each of the ontologies from which it references classes. Only in one case did the more expressive reasoner reveal additional unsatisfiable classes, in the case of the Ontology of Biomedical Investigations (OBI) (Brinkman et al., 2010) . Most of our tests with more expressive reasoners were unsuccessful, because the reasoner was not able to complete classification of the ontology. This was either due to the undecidable nature of reasoning over ontologies in OWL Full, or due to the complexity required to reason over ontologies in OWL 2 DL. This result highlights one of the major advantages of MIREOT, especially when building ontologies in a situation where more expressive axioms are required.

After detecting this large number of unsatisfiable classes, we examined some of the problematic axioms which have arisen due to the combination of the ontologies. For example, Figure 1 demonstrates that the root cap (PO:0020123) class from the Plant Ontology (PO (The Plant Ontology Consortium, 2002) has become unsatisfiable when combined with EFO. In the PO, an axiom states that portion of plant tissue must be disjoint from the cardinal organ part, and that the root cap is a subclass of portion of plant tissue. Additionally, the PO states that root cap is a part of some root tip, which is a subclass of cardinal organ part. However, in EFO, the reference to root cap asserts that it is a direct subclass of root tip – a subclass of cardinal organ part. This assertion causes root tip to be a subclass of both portion of plant tissue and of cardinal organ part, an assertion which violates the disjointness axiom between them. The correct superclass of root tip, provided by PO, is root parenchyma, and asserting only this subclass axiom would prevent the resulting unsatisfiability. Figure 1 also demonstrates the “downstream” effect which this assertion causes as all sub-classes of an unsatisfiable class (i.e., root cap) also become unsatisfiable. Therefore, many unsatisfiable classes from Table 2 may be the consequence of relatively few unsatisfiable classes close to the root of the ontologies’ class hierarchy.

In another case, we observed that UBERON includes an axiom stating respiratory system must be disjoint with digestive system. The chordate pharynx class, in UBERON, is both a subclass of part of some digestive system and part of some respiratory system. However, when the EFO references this class it asserts that it is a subclass of both the digestive system and respiratory system directly. In this case, one class is restricted in EFO to be a subclass of two classes which directly contradict each other. The unsatisfiability would not occur if the class reference used the taxonomical superclass – the pharynx. Alternatively, the contradiction could be directly observed and avoided during development, if the full UBERON ontology was imported.

It seems that, both in the case of the root cap and the chordate pharynx, unsatisfiability has occurred due to the addition of axioms that directly violate disjointness axioms in the referenced ontologies. In particular, axioms involving object properties (i.e., parthood relations) seem to be expressed as direct subclass relationships.

Violation of disjointness can result in complex errors which may be difficult to trace in the original ontologies. For example, in EFO, it is asserted that blastocyst (UBERON:0000358) is the direct subclass of both blastula stage (UBERON:0000108) and embryonic structure (UBERON:0002050); blastula stage in UBERON is a subclass of processual entity. The blastula stage is an abstract staging of development, and while not used to describe the stage of mammalian development immediately after the morula in modern scientific discourse, it is a useful landmark stage for most multicellular embryos, and describes the acquisition of an internal cavity in the morula, the blastocoel cavity. Blastocyst refers to a physical entity in mammalian development and so the EFO here breaks a disjointness axiom asserted in UBERON between anatomical entity and processual entity. In UBERON, there is no assertion that blastocyst is a subclass of blastula stage, only that it is a subclass of embryonic structure. Through the application of MIREOT, the disjointness between processual and anatomical entities in UBERON is lost and the additional axiom has been added that blastocyst is a subclass of blastula stage (likely as a replacement of an axiom involving the existence-starts-and-ends-during object property in UBERON). Application of the MIREOT method has again hidden this contradiction.

Finally, we also test the consistency of EFO combined with all ontologies from which classes are referenced, i.e., we import all ontologies in Table 2 and use an OWL 2 EL reasoner to classify the resulting ontology. We find that EFO combined with all these ontologies is inconsistent. The cause of the inconsistency lies in the Eagle-i Resource Ontology (ERO) (Torniai et al., 2011) , in which the realized in object property has been obsoleted (i.e., made a subproperty of obsolete object property), and the domain and range of obsolete object property has been declared as owl:Nothing. This inconsistency is therefore likely the result of a versioning problem in which EFO was built using an older version of the ERO, and the newer version of ERO has evolved to become incompatible with EFO.

We then removed the ERO ontology from the set of ontologies we add to EFO, and classified the remaining set of ontologies. This approach removed the inconsistency, but resulted in a total of 52,540 unsatisfiable classes out of a total of 297,591 classes in the combined ontology, around 1/6th of all classes. 3.2

The unMIREOT tool

To automate and generalize our experiments, we have developed the unMIREOT tool. unMIREOT can be applied to any ontology for which the MIREOT method has been used. unMIREOT utilises AberOWL (Hoehndorf et al., 2015) and the OWLAPI (Horridge et al., 2007) to acquire the authoritative ontologies from which classes are used, and then produces a report on whether unsatisfiable classes are found when importing the referenced ontologies. The unMIREOT tool can be found at https://github.com/ bio-ontology-research-group/UNMIREOT/.

One major limitation with the unMIREOT tool is that it currently only works with ontologies available through the AberOWL repository, and that it performs only a single unMIREOT step. Specifically, if an ontology O1 is referenced in an ontology O, unMIREOT will import the axioms of O1 into O and generate a report based on the combined ontology; however, if O1 itself also used the MIREOT method and references ontology O2, the axioms of O2 would not be included in the combined ontology. In the future, unMIREOT will be extended to transitively unMIREOT ontologies. 4

DISCUSSION

MIREOT, and the reuse of class IRIs across ontologies, indubitably has great advantages. Whenever a class is created that is intended to represent something that already exists in another ontology, and is well-defined and axiomatized in that ontology, this class can be reused without duplicating the effort. Should it be required to combine two ontologies, proper application of the MIREOT principles (in particular the inclusion of the source ontology’s IRI) will ensure that the axioms can be merged and knowledge in two or more ontologies combined without the additional steps of mappings classes. More importantly, use of the MIREOT method makes ontology development feasible with limited computational resources and tool support that make developing large and complex ontologies prohibitively expensive. However, there is also the temptation to develop an ontology, using classes from other ontologies without explicitly verifying whether the developed ontology is consistent and coherent with respect to the axioms that constrain the referenced classes in their original ontology; consequently, additional axioms may be introduced that directly or indirectly contradict the intended meaning of the class in the source ontology.

We have observed that it is indeed the case with EFO that classes are used “out of context” and given a new meaning that is local to EFO, yet is logically contradictory with the axioms constraining the class in its source ontology. Within the AberOWL repository, over 150 ontologies currently use MIREOT or a similar method, reusing classes from other ontologies without importing them, and the consistency problems we identified do not only occur in EFO, but are also present in other ontologies. Preliminary results for the Ontology of Biomedical Investigations (OBI) (Bandrowski et al., 2016) can be found at https://github. com/bio-ontology-research-group/UNMIREOT, and we intend to perform a similar analysis for all ontologies in AberOWL as future work.

Another factor which affects the continued compatibility of an ontology with the ontologies it references is the time at which a reference to a class was made. Our results show that in some cases, the unsatisfiabilities stem from the fact that referenced classes or parent classes of such have since changed, or become obsolete, in the referenced ontology. This highlights the need for improved versioning, and including the exact date and original ontology based on which a MIREOT was performed, and provides a continuing challenge to keep ontologies up-to-date with other related ontologies.

Our findings show that the unconstrained use of the MIREOT method may have introduced a new challenge for ontology interoperability, which must now be addressed, in particular if unsatisfiable classes and inconsistent ontologies are found on a wider scale with additional and more extensive tests on additional ontologies besides EFO. The question remains, however, how best to balance the challenges of developing ontologies with the hardware resources and tools available, while at the same time maintaining consistency and interoperability between ontologies.

One approach to preventing inconsistencies, incompatibilities and unsatisfiable classes could be the inclusion of a validation stage in which ontology creators who use MIREOT ensure the interoperability and mutual consistency of their developed ontology with the ontologies it references before every release. As a starting point, to help with these tests, we have developed the unMIREOT tool; unMIREOT can be used to fully import all referenced ontologies, determine the consistency of the resulting ontology, then discover and list any contradictions which may have been introduced. If no contradictions were found, the ontology could then be released, either with all import statements in place or using the MIREOT method as intended. Initially, the unMIREOT tool may also be used to explore and correct currently existing incompatibilities between biomedical ontologies. Further development of unMIREOT could potentially allow for a more fine-grained output that encompasses the nature of the unsatisfiability, providing detailed explanations which would assist ontology creators in resolving them.

ACKNOWLEDGMENTS

LTS and GVG were supported by the National Science Foundation (grant IOS:1340112). RH was supported by funding from the King Abdullah University of Science and Technology.

References

Ashburner , M. , Ball , C. A. , Blake , J. A. , Botstein , D. , Butler , H. , Cherry , M. J. , Davis , A. P. , Dolinski , K. , Dwight , S. S. , Eppig , J. T. , Harris , M. A. , Hill , D. P. , Tarver , L. I. , Kasarskis , A. , Lewis , S. , Matese , J. C. , Richardson , J. E. , Ringwald , M. , Rubin , G. M. , and Sherlock , G. ( 2000 ). Gene ontology: tool for the unification of biology . Nature Genetics , 25 ( 1 ), 25 - 29 .

Bandrowski , A. , Brinkman , R. , Brochhausen , M. , Brush , M. H. , Bug , B. , Chibucos , M. C. , Clancy , K. , Courtot , M. , Derom , D. , Dumontier , M. , Fan , L. , Fostel , J. , Fragoso , G. , Gibson , F. , Gonzalez-Beltran , A. , Haendel , M. A. , He , Y. , Heiskanen , M. , Hernandez-Boussard , T. , Jensen , M. , Lin , Y. , Lister , A. L. , Lord , P. , Malone , J. , Manduchi , E. , McGee , M. , Morrison , N. , Overton , J. A. , Parkinson , H. , Peters , B. , Rocca-Serra , P. , Ruttenberg , A. , Sansone , S.-A. , Scheuermann , R. H. , Schober , D. , Smith , B. , Soldatova , L. N. , Stoeckert , Jr., C. J. , Taylor , C. F., Torniai , C. , Turner , J. A. , Vita , R. , Whetzel , P. L. , and Zheng , J. ( 2016 ). The ontology for biomedical investigations . PLoS ONE , 11 ( 4 ), 1 - 19 .

Brinkman , R. R. , Courtot , M. , Derom , D. , Fostel , J. M. , He , Y. , Lord , P. , Malone , J. , Parkinson , H. , Peters , B. , Rocca-Serra , P. , Ruttenberg , A. , Sansone , S.-A. , Soldatova , L. N. , Stoeckert , Jr, C. J. , Turner , J. A. , Zheng , J. , and , O. B. I. c. ( 2010 ). Modeling biomedical experimental processes with obi . J Biomed Semantics , 1 Suppl 1 , S7 .

Courtot , M. , Gibson , F. , Lister , A. L. , Malone , J. , Schober , D. , Brinkman , R. R. , and Ruttenberg , A. ( 2011 ). Mireot: The minimum information to reference an external ontology term . Applied Ontology , 6 ( 1 ), 23 - 33 .

Degtyarenko , K. , Matos , P. , Ennis , M. , Hastings , J. , Zbinden , M. , McNaught , A. , Alcantara , R. , Darsow , M. , Guedj , M. , and Ashburner , M. ( 2007 ). ChEBI: a database and ontology for chemical entities of biological interest . Nucleic Acids Research .

Grau , B. , Horrocks , I. , Motik , B. , Parsia , B. , Patelschneider , P. , and Sattler , U. ( 2008 ). OWL 2: The next step for OWL . Web Semantics: Science, Services and Agents on the World Wide Web , 6 ( 4 ), 309 - 322 .

Hoehndorf , R. , Dumontier , M. , and Gkoutos , G. V. ( 2013 ). Evaluation of research in biomedical ontologies . Briefings in Bioinformatics , 14 ( 6 ), 696 - 712 .

Hoehndorf , R. , Slater , L. , Schofield , P. N. , and Gkoutos , G. V. ( 2015 ). Aber-OWL: a framework for ontology-based data access in biology . BMC Bioinformatics , 16 , 26 .

Horridge , M. , Bechhofer , S. , and Noppens , O. ( 2007 ). Igniting the OWL 1.1 touch paper: The OWL API . In Proceedings of OWLED 2007: Third International Workshop on OWL Experiences and Directions.

Kapushesky , M. , Emam , I. , Holloway , E. , Kurnosov , P. , Zorin , A. , Malone , J. , Rustici , G. , Williams , E. , Parkinson , H. , and Brazma , A. ( 2010 ). Gene expression atlas at the european bioinformatics institute . Nucleic Acids Research , 38 ( suppl 1 ), D690 - D698 .

Kazakov , Y. , Kro¨tzsch, M., and Simancik , F. ( 2014 ). The incredible elk . Journal of Automated Reasoning , 53 ( 1 ), 1 - 61 .

Malone , J. , Holloway , E. , Adamusiak , T. , Kapushesky , M. , Zheng , J. , Kolesnikov , N. , Zhukova , A. , Brazma , A. , and Parkinson , H. ( 2010 ). Modeling sample variables with an experimental factor ontology . Bioinformatics , 26 ( 8 ), 1112 - 1118 .

Motik , B. , Shearer , R. , and Horrocks , I. ( 2009 ). Hypertableau Reasoning for Description Logics . Journal of Artificial Intelligence Research , 36 , 165 - 228 .

Mungall , C. , Torniai , C. , Gkoutos , G. , Lewis , S. , and Haendel , M. ( 2012 ). Uberon, an integrative multi-species anatomy ontology . Genome Biology , 13 ( 1 ), R5 .

Noy , N. F. , Sintek , M. , Decker , S. , Crubezy , M. , Fergerson , R. W. , and Musen , M. A. ( 2001 ). Creating semantic web contents with Protege-2000 . IEEE Intelligent Systems , 16 ( 2 ), 60 - 71 .

Parkinson , H. , Sarkans , U. , Kolesnikov , N. , Abeygunawardena , N. , Burdett , T. , Dylag , M. , Emam , I. , Farne , A. , Hastings , E. , Holloway , E. , Kurbatova , N. , Lukk , M. , Malone , J. , Mani , R. , Pilicheva , E. , Rustici , G. , Sharma , A. , Williams , E. , Adamusiak , T. , Brandizi , M. , Sklyar , N. , and Brazma , A. ( 2011 ). Arrayexpress update-an archive of microarray and highthroughput sequencing-based functional genomics experiments . Nucleic Acids Research , 39 ( suppl 1 ), D1002 - D1004 .

Sarntivijai , S. , Vasant , D. , Jupp , S. , Saunders , G. , Bento , A. P. , Gonzalez , D. , Betts , J. , Hasan , S. , Koscielny , G. , Dunham , I. , Parkinson , H. , and Malone , J. ( 2016 ). Linking rare and common disease: mapping clinical disease-phenotypes to ontologies in therapeutic target validation . J Biomed Semantics , 7 , 8 . Sarntivijai, Sirarat Vasant, Drashtti Jupp, Simon Saunders, Gary Bento , A Patricia Gonzalez , Daniel Betts, Joanna Hasan, Samiul Koscielny, Gautier Dunham, Ian Parkinson, Helen Malone, James eng Research Support, Non-U.S. Gov' t England 2016 /03/25 06:00 J Biomed Semantics . 2016 Mar 23 ; 7:8 . doi: 10 .1186/s13326-016-0051-7. eCollection 2016 .

Sayers , E. W. , Barrett , T. , Benson , D. A. , Bryant , S. H. , Canese , K. , Chetvernin , V. , Church , D. M. , DiCuccio , M. , Edgar , R. , Federhen , S. , Feolo , M. , Geer , L. Y. , Helmberg , W. , Kapustin , Y. , Landsman , D. , Lipman , D. J. , Madden , T. L. , Maglott , D. R. , Miller , V. , Mizrachi , I. , Ostell , J. , Pruitt , K. D. , Schuler , G. D. , Sequeira , E. , Sherry , S. T. , Shumway , M. , Sirotkin , K. , Souvorov , A. , Starchenko , G. , Tatusova , T. A. , Wagner , L. , Yaschenko , E. , and Ye , J. ( 2009 ). Database resources of the national center for biotechnology information . Nucleic Acids Research , 37 ( suppl 1 ), D5 - D15 .

Smith , B. , Ashburner , M. , Rosse , C. , Bard , J. , Bug , W. , Ceusters , W. , Goldberg , L. J. , Eilbeck , K. , Ireland , A. , Mungall , C. J. , Leontis , N. , Serra , P. R. , Ruttenberg , A. , Sansone , S. A. , Scheuermann , R. H. , Shah , N. , Whetzel , P. L. , and Lewis , S. ( 2007 ). The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration . Nat Biotech , 25 ( 11 ), 1251 - 1255 .

Smith , C. L. , Goldsmith , C.-A. W. , and Eppig , J. T. ( 2004 ). The mammalian phenotype ontology as a tool for annotating, analyzing and comparing phenotypic information . Genome Biol , 6 ( 1 ), R7. DOI: 10 .1186/gb-2004-6-1-r7.

The Plant Ontology Consortium ( 2002 ). The plant ontology consortium and plant ontologies . Comparative and Functional Genomics , 3 ( 2 ), 137 - 142 .

Torniai , C. , Brush , M. H. , Vasilevsky , N. , Segerdell , E. , Wilson, M. , Johnson , T., Corday , K. , Shaffer , C. , and Haendel , M. ( 2011 ). Developing an application ontology for biomedical resource annotation and retrieval: Challenges and lessons learned . In Proceedings of the 2nd International Conference on Biomedical Ontology , Buffalo, NY , USA, July 26 - 30 , 2011 .

Tsarkov , D. and Horrocks , I. ( 2006 ). FaCT++ description logic reasoner: System description . Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 4130 LNAI, 292 - 297 .