Two important characteristics of science are the “reproducibility” and “clarity”. By rigorous practices, scientists explore aspects of the world that they can reproduce under carefully controlled experimental conditions. The clarity, complementing reproducibility, provides unambiguous descriptions of results in a mechanical or mathematical form. Both pillars depend on well-structured and accurate descriptions of scientific practices, which are normally recorded in experimental protocols, scientific workflows, etc. Here we present SMART Protocols (SP), our ontology-based approach for representing experimental protocols and our contribution to clarity and reproducibility. SP delivers an unambiguous description of processes by means of which data is produced; by doing so, we argue, it facilitates reproducibility. Moreover, SP is thought to be part of e-science infrastructures. SP results from the analysis of 175 protocols; from this dataset, we extracted common elements. From our analysis, we identified document, workflow and domain-specific aspects in the representation of experimental protocols. The ontology is available at http://purl.org/net/SMARTprotocol
Scientific experiments often bring together several technologies at in vivo, in vitro and sometimes in silico levels. Moreover, the biomedical domain relies on complex processes, comprising hundreds of individual steps usually described in experimental protocols. An experimental protocol is a sequence of tasks and operations executed to perform experimental research. The protocols often include equipment, reagents, critical steps, troubleshooting, tips and all the information that facilitates reusability. Researchers write the protocols to standardize methods, to share these documents with colleagues and to facilitate the reproducibility of results.
Although reproducibility, central to research, depends on well-structured and accurately described protocols, scientific publications often lack sufficient information when describing the protocols that were used. For instance, there is ambiguity in the terminology as well as poor descriptions embedded within a heterogeneous narrative. There is the need for a unified criterion with respect to the syntactic structure and the semantics for representing experimental protocols.
Here we present SMART Protocols (henceforth SP), our ontology-based approach
for representing experimental protocols. SP aims to formalize the description of
experimental protocols, which we understand as domain-specific workflows
embedded within documents. SP delivers a structured workflow, document and
domain knowledge representation written in OWL DL. For the representation of
document aspects we are extending the Information Artifact Ontology (IAO).1 The
representation of executable aspects of a protocol is captured with concepts from
PPlan Ontology (P-Plan) [
Our main assumption is that “experimental protocols are fundamental information structures that should support the description of the processes by means of which results are generated in experimental research”. Hence our approach should allow answering questions such as: Who is the author of the protocol? What is the application of the protocol? What are the reagents, equipment and/or supplies used? What is the estimated time to execute a protocol? Which samples have been tested in a protocol? This paper is organized as follows: Section 2 presents related works, Section 3 describes the methodology stages to develop the SP ontology, section 4 shows the results and ontology evaluation. Finally Section 5 provides discussion and conclusions.
In an effort to address the problem of inadequate methodological reporting, the
MIBBI2 project brings under one umbrella most of these projects. The ISA-TAB also
illustrates work in this area; it delivers metadata standards to facilitate data collection,
management and reuse [
The Ontology for Biomedical Investigations (OBI)3 aims to model the design of investigations, including the protocols, materials used and the data generated. OBI has key classes for the description of experiments, namely: obi:investigator, obi:instrument, obi:biomaterial entity. The generic ontology of scientific experiments (EXPO)4 aims to formalize domain-independent knowledge about the planning, execution and analysis of scientific experiments. This ontology includes the class expo:ExperimentalProtocol and defines some of its properties: expo:has_applicability, expo:has_goal, 1 https://code.google.com/p/information-artifact-ontology/ 2 http://mibbi.sourceforge.net/portal.shtml 3 http://obi-ontology.org/page/Main_Page 4 http://expo.sourceforge.net/ expo:has_plan. EXACT suggests a meta-language for the description of experiment actions and their properties.
Recently, PLOS ONE in collaboration with Science Exchange and Figshare launched “The Reproducibility Initiative”.5 This project aims to help scientists to validate their research findings. The Research Object initiative6 aims to deliver a model to represent experimental resources; this model facilitates accessibility, reusability, reproducibility and also a better understanding of in silico experiments.
Publishers are also actively addressing the problem of experimental
reproducibility; F1000Research,7 an open science journal, suggests data preparation
guidelines to capture the processes and procedures required to publish scientific
dataset. The Force 11 initiative,8 a community of researchers addressing issues in
scholarly communication, has published a set of metadata standards for biomedical
research. These standards focus on three recommendations: Gene accession numbers,
organism identification and reagent identification. Vasilevsky et al., [
Unlike other approaches, the SP ontology provides a formalized representation of the domain that is not sufficiently covered by other ontologies. For instance, SPdocument delivers a structured vocabulary representing a specific type of document, a protocol. This vocabulary includes rhetorical components (e.g. introduction, materials, and methods); it also has information like application of the protocol, advantages and limitations, list of reagents, critical steps. In addition, The formalization of instructions in the protocol, or steps, is covered in SP-workflow by the class p-plan:Step. The order in which these steps should be executed is captured by the property bfo:isPrecededBy. Inputs and outputs from each step are represented by the class p-plan:Variable. 2
For designing SP, we followed practices recommended by the NeOn methodology
[
In this stage we identified reusable terminology from other ontologies; supporting
activities throughout this stage we used BioPortal.9 We also looked into minimal
information standards,10 guidelines and vocabularies representing research activities
[
We manually reviewed 175 published and non-published protocols from domains like biotechnology, virology, biochemistry and pathology. The non-published protocols (75 in total) were collected from four laboratories located at International Center for Tropical Agriculture (CIAT).11 The published protocols (open access protocols in plant biology) were gathered from 9 repositories: Biotechniques, 12 Cold Spring Harbor Protocols (CSH Protocols), 13 Current Protocols (CP), 14 Genetics and Molecular Research (GMR),15 Journal of Visualized Experiments (JoVE),16 Protocol Exchange (PE),17 Plant Methods (PM),18 Plos One (PO)19 and Springer Protocols (SP)20 (Table 1).
throughout our development process. We identified linguistic structures that authors were using to represent actions; we needed to understand how instructions were organized. We were interested in understanding how verbs were representing actions, what additional information was there for indicating attributes for actions. By analyzing texts we were also identifying terminology and determining whether these terms were already available in existing ontologies. Minimal information standards were also considered; how could these be used when describing an experimental protocol? 11 http://ciat.cgiar.org/ 12 http://www.biotechniques.com/protocols/ 13 http://cshprotocols.cshlp.org/ 14 http://www.currentprotocols.com/WileyCDA/ 15 http://www.geneticsmr.com/ 16 http://www.jove.com/ 17 http://www.nature.com/protocolexchange/ 18 http://www.plantmethods.com/ 19 http://www.plosone.org/ 20 http://www.springerprotocols.com/ 21 http://goo.gl/MC4mR9 From our dataset we extracted common elements and evaluated whether those protocols could be implemented. Initially, we focused our analysis on identifying necessary and sufficient information for reporting protocols. From our inspection, we determined workflow aspects in experimental protocols. The sequence of instructions had an implicit order, following the input output structure. Actions in the workflow of instructions were usually indicated by verbs; accurate information for implementing the action implicit in the verb was not always available. For instance, structures such as “Mix thoroughly at room temperature”, “Briefly spin the racked tubes” are common in our dataset. Due to the ambiguity and lack of detailed information for specifying actions in the instructions, it was difficult to understand how could these be implemented. Domain expertise was usually required in order to interpret some of the actions in our dataset.
In addition, we also isolated elements pertaining to domain knowledge as well as document related characteristics. We classified our protocols within 4 groups according to the purpose, namely: i) plant genetic transformation, ii) DNA/RNA extraction and purification, iii) PCR and their variants, iv) electrophoresis and sequencing. Within each group we identified basic steps (or common patterns), which we consider as necessary in the structure of the protocol. For example, we found that a cell disruption step is essential in DNA extraction protocols. We also identified that a digestion reaction (removing the lipid membrane, proteins and RNA) follows and that the DNA precipitation or purification comes at the end of this process.
Variables Cell disruption (CD) Digestion reaction (DR) DNA precipitation or purification (DNAP)
Constants First= first step Second= second step
Third= third step dna_extraction_protocol(CD, DR, DNAP):—
CD= first, DR= second, DNAP= third 3. Iterative ontology building and validation, IO: as we were gathering information and learning about this domain, we started by building concept maps; these were rapidly mapped to parts of speech from the texts we were analyzing and also to existing ontologies. As concept maps were growing in complexity, number of concepts and relations, we then started to build draft ontologies –baseline ontologies representing specifics from parts of speech we identified. The knowledge engineer conducted the evaluation of the draft ontologies against competency questions. Models were also exchanged with domain experts; the process was iterative and, the models were constantly growing.
By building ontology models as well as by carefully analyzing the information we were gathering from LISA and DAKA activities, we were able to identify the modularity needed to represent experimental protocols. The module SP-document was designed to provide a structured vocabulary of concepts to represent information for recording and reporting an experimental protocol. The module SP-workflow aims to provide a structured vocabulary of concepts to represent the execution of experimental protocols in life sciences. 2.3
The goal of the evaluation is to determine what the ontology defines, and how
accurate these definitions are. Here we follow the activities proposed by Gómez-Pérez
et al. [
At the end of the cycle, new classes, properties and individuals are identified. These are then analyzed against the set to competency questions, existing ontologies, parts of speech and linguistic structures. The model evolves as new knowledge goes through the whole cycle. Although our ontology is a young ontology, we could observe how it evolved in its conceptualization as well as in the explicit specification. 22 http://www.w3.org/TR/2004/REC-owl-features-20040210/#s1.3 23 http://protegewiki.stanford.edu/wiki/OWLViz 24 http://oeg-lia3.dia.fi.upm.es/oops/index-content.jsp 25 goo.gl/jBHPo 26 goo.gl/gAVnn
The SMART Protocols approach follows the OBO Foundry principles [
An overview of the two modules comprising SP is illustrated in Figure 2. The classes, properties and individuals are represented by their respective labels to facilitate the readability. The prefix indicates the provenance of each term. The ontology describing the experimental protocol as a document is depicted at the top. The class iao:information content entity and its subclasses iao:document, iao:document part, iao:textual entity and iao:data set were imported from The Information Artifact Ontology (IAO) to represent the document aspects in the protocol. The ontology describing the experimental protocol as a workflow is depicted at the bottom. The representation of executable aspects of a protocol is modeled with the classes p-plan:Plan, pplan:Step and p-plan:Variable from the P-Plan Ontology (P-Plan). 27 http://www.ifomis.org/bfo/ 28 http://obi-ontology.org/page/OBI_Minimal_metadata 3.1
The document module of SMART Protocols reuses classes from CHEBI [
Use Case. SP-document represents information such as, the protocol type, sp:DNA
extraction protocol; it has a tittle, identified by the property sp:has
title, it is instantiated by genomic DNA isolation. Also, the author entry,
iao:author identification, is instantiated by CIMMYT [
Use Case. DNA extraction is a procedure frequently used to collect DNA for subsequent molecular or forensic analysis. DNA extraction includes 3 basic pplan:Steps: i) cell disruption or cell lysis, ii) Digestion reaction (in this step, contaminants such as lipid membrane, proteins and RNA are removed from the DNA solution), and iii) DNA purification. Each one of these steps may include different protocols (or p-plan:Plans) to be executed. For example, the step sp:cell disruption or cell lysis may be achieved by chemical and physical methods blending, grinding or sonicating the sample. Also, the ontology considers that each step is executed following a predetermined order. For instance, according to the protocol published by CIMMYT, the cell disruption by lyophilization and grinding has an input variable, p-plan:hasInputVar, as well as sp:plant tissue; it also has an output, p-plan:hasOutputVar, and sp:powdered tissue. The next step, sp:digestion reaction, has as input the output of the immediately previous step, sp:powdered tissue, and as output sp:digested contaminant. The last one, sp:DNA purification has as input sp:digested contaminant, and as output obi:DNA extract (Fig. 4).
Science has, among other, two important characteristics, reproducibility and clarity.30 Clarity provides unambiguous descriptions for results in a mechanical or mathematical form. The lack of clarity about "how to do or how to execute" an experimental procedure hinders the reproducibility and impedes comparing results across related experiments. SMART Protocols addresses clarity by formalizing the objects that should go together with actions. Besides, SP reuses and extends minimal information standards, incorporating these structures within the representation of experimental protocols. By delivering a semantic and syntactic structure SP also facilitates reproducibility.
Our ontology-based representation for experimental protocols is composed of two
modules, namely SP-document and SP-workflow. In this way, we represent the
workflow, document and domain knowledge implicit in experimental protocols. Our
work extends IAO and P-Plan ontology. Actions, as presented by [
Formalizing workflows has an extensive history in Computer Science; not only in planning but also in execution –as in Process Lifecycle Management and Computer Assisted Design/Computer Assisted Manufacturing. We have considered some of these principles for representing workflow aspects in protocols just as we have reused knowledge formalisms like the P-Plan Ontology. By formalizing the workflow implicit in protocols the execution can be ontologically represented in a sequential manner that is intelligible by humans and processed by machines.
Modularization, as it has been implemented in SP, facilitates managing the ontology. For instance, the workflow module can easily be specialized with more specific formalisms so that robots can process the flow of tasks to be executed. The document module facilitates archiving; the structure also allows to have fully identified reusable components. By combining both modules we are delivering a selfdescribing document. 30 http://www.aaas.org/page/so-can-science-explain-everything