Challenges and opportunities for system biology standards and tools in medical research

Abstract

Kinetic models are increasingly relevant in medical research. In systems biology, more than 10 years of experience with the development of standards and tools to construct and analyse kinetic models exists. This has supported the sharing of kinetic models, increased their reuse, and thereby has helped to reproduce and validate scientific results. Given this expertise, it seems natural to consider the application and development of standards and tools to meet the requirements of medical scientists. In this paper, we discuss challenges and opportunities for standards and tools from systems biology in medical research, and we suggest criteria for the safe use of simulations. We conclude that standards, tools and infrastructure need to be extended to ensure the quality, reliability and safety required when working with medical and patient data. This will foster the adaptation of modelling in the clinic, providing tools for improved diagnosis, prognosis and therapy. Contact: [e-mail]

INTRODUCTION

In modern medicine, technologies complement conventional clinical data with molecular and genetic information. Patient-specific molecular profiling provides opportunities for earlier diagnosis, more accurate prognoses and optimised therapeutic decisions [ 1 ]. The data generated from these new technologies have led to a rise of computational approaches in medicine [ 2 ].

‘Personalised Medicine’ and ‘Systems Medicine’ are two terms that are frequently used to capture this trend for interdisciplinary approaches in which clinical research, molecular and cell biology, medical informatics, bioinformatics, biostatistics and systems biology approaches join forces. Personalised medicine uses markerassisted diagnosis and targeted therapies derived from an individual’s molecular profile and patient data [ 3 ]. Systems medicine aims to bring computational models closer to the clinic to shed light on the dynamic complexity of human physiology and disease [ 4 ]. In this context, the focus has been on the modelling of phenomena, where an understanding of processes (kinetics) is crucial. This includes the response of cells, tissues and organs to drugs [ 5 ]; the to whom correspondence should be addressed simulation of disease progression [ 6 ]; and the understanding of mechanisms, as opposed to just predicting outcomes [ 7 ].

With new technologies available to provide the data to identify and characterise disease relevant components, there is an increasing demand for methodologies that enable us to study the interactions of molecular and cellular components in a patient. Arguably, the success of systems and personalised medicine relies then on the application of kinetic models in the clinic [ 8 ].

The construction of such models requires an integration of clinical and patient-specific molecular data with public databases such as Ensembl [ 9 ] and ENCODE [ 10 ]. This process effectively brings together the two worlds of basic research and clinical practice. For this union to succeed, ontologies will play a crucial role. Standards to encode information together with ontologies to unambiguously characterise domain knowledge, form the basis for the development of tools that can analyse kinetic models. These tools in turn support the sharing and reuse of models, which is also a means to validate results and generally improve reproducibility in medical research.

Here we illustrate the challenges that need to be overcome in future work to achieve trustworthy systems that can be integrated easily into a clinical environment. The structure of the remaining sections of this paper is as follows. In section 2 , we outline the challenges that exist when planning to use state-of-the-art systems biology tools and standards in clinical environments. Following on from that, in section 3 we suggest criteria that address the challenges outlined and need to be taken into consideration when building clinically applicable solutions. We present a summary of our findings in the last section of this paper. 2 2.1

CHALLENGES IN APPLYING SYSTEMS

BIOLOGY STANDARDS AND TOOLS

Access to clinical data

Almost no clinical data sets are available for integration with models, neither are these data sets sufficiently documented in a formalised manner. Consequently, the process of selecting clinical data for a given model (and vice versa) is hindered. This is partly due to patient data being sensitive, limiting its accessibility for analysis, but mainly due to missing incentives, guidelines and requirements to provide data access upon publication of clinical studies.

Clinical data sets are required for testing as well as prediction purposes. While a reoccurring complaint is the lack of suitable data sets to test a model with, this problem is hard to overcome given that patient data needs to be secured over unauthorised access at all times or anonymised in a proper manner. Some efforts such as the 100,000 genome project conducted by Genomics England1 and the openEHR2 project aim to provide access to structured, semantically annotated clinical data for research purposes. However, the amount of available data is still too limited to test models and computational simulations reliably.

In practice, most research data are neither shared nor recycled outside the original project team [ 11 ]. Models are instead being developed and used within a single clinic, e. g. by collaborative projects that incorporate clinical research groups and computational biology groups located in the same institution. In these settings, however, modelling has already been applied successfully, for example to study melanoma resistance to immunotherapy [ 12 ]. 2.2

Good quality models and documentation

In addition to relevant clinical data being accessible, it must be represented in a way that it can be integrated and interpreted by both humans and machines. This requires a dialogue not only between healthcare providers and researchers, but also with staff recording the data and policy makers regulating patient data records.

Currently, the majority of published models are not available in standard formats, and the model quality is not sufficiently documented. While promoting the reuse of such virtual experiments would vastly improve the usefulness and relevance of computational models in biomedical endeavours [ 13 ], even the computational code underlying a model is often inaccessible. Without the ability of reproducing the models, however, models cannot be exploited for clinical use. SED-ML is a standard for the encoding of simulation setups, the specification of possible parametrisations and the definition of analyses [ 14 ]. However, SED-ML to date encodes only for a subset of experiments performed in clinical research. Further extensions are needed in the standard itself.

In addition, available models are not fully annotated, i.e. the description of model components and parameters are missing, hindering interpretation and integration with other models and clinical data. Model provenance information is not kept, leading to misinterpretations and even irreproducibility of the original findings.

Ongoing efforts such as curation processes in BioModels3, or the provision of fully reproducible archives of virtual experiments in the Physiome Model Repository [ 15 ] or in the JWS Online database [ 16 ] improve this situation. However, curation is very slow due to the manual labour involved and seldom performed after a model has been published. Moreover, concerted efforts for model validation, annotation, and conversion into computable formats are lacking. 2.3

Standardised representation of models and data

The systems biology community developed a set of interoperable standards for modelling in biology, including the Systems Biology Markup Language (SBML), CellML, Synthetic Biology Markup Language (SBOL), NeuroML, Simulation Experiment Description

1 https://www.genomicsengland.co.uk/

the-100000-genomes-project/

2 http://www.openehr.org

3 http://www.ebi.ac.uk/biomodels-main/

Markup Language (SED-ML), or BioPax [ 17 ]. As a consequence, sharing and/or integrating models within communities is feasible. However, model reuse across communities can be challenging, as different standards are used for the representation and annotation of the data.

Even within communities, there is no consensus on which ontologies to use for data and model representation. It is also not defined to which degree of detail models and data need to be annotated, creating further obstacles to integrate models for simulation purposes. Extensive cross-domain initiatives need to be built and are required to take decisions on ontologies and standards that are not only convenient for model developers, curators and researchers, but that are also practical (implementation, costs, etc.) in a clinical application scenario. 2.4

Validated predictions in a clinical context

A major hurdle for the translation of computational models into medical research is the difficulty to proof the efficiency and predictive value of the model. Every recommendation determined by a clinical decision support system needs to be in line with the policies for medical care providers as issued by the health authorities in the respective country. In order to proof health economic efficiency, extensive, potentially double-blinded, clinical trials are required that compare model-based treatment decisions with unsupported decisions by clinical staff. These clinical trials have to span over all areas of clinical application, i.e. cover different types of diseases as well as ranges of treatments and patients in differing health conditions to assess clinical safety. Every in silico model provides an estimation of pathological processes and therefore naturally contains errors. These errors can potentially lead to wrong treatment decisions, which is why great care needs to be taken when transporting systems biology models, standards and tools into clinical practice. Sustained software support is equally important. Software libraries for standards should to be stable, well-tested, and they should support the complete standard in correct manner. Such implementations will facilitate the update of standards by the community and tool developers and thus provide shareable data and models. 3

CRITERIA FOR REUSABLE SIMULATION

MODULES AND SEMANTIC DATA

The reproducibility and reusability of models and model-based results have been discussed in several assays over the past years [ 8, 18 ]. One conclusion of these assays is that the reusability of simulation models needs to be ensured, before computational models can be considered for predictive processes in the clinic. Four important aspects that determine reusability are discussed in the following subsections. 3.1

Semantic annotation via biomedical ontologies

An essential step to ensure reusability of models is a thorough semantic annotation to biomedical ontologies. An ontology formally defines concepts and relations between concepts in a knowledge domain [ 19 ]. In the context of this paper, semantic annotation describes the process of linking the entities and processes of a model to terms in relevant ontologies. These semantic descriptions allow researchers and tools alike to describe the data used in experimental studies and models. They enable not only the integration of different types of data but also the reasoning over the data, thus connecting data items (or models) to existing knowledge. Systems biology established a system for semantic annotations of models, using RDF together with standardised relationships [ 20 ] and resources identifiers [ 21 ]. Recently, composite annotations have been proposed as a means to provide exact descriptions of the model entities [ 22 ].

In order to implement models in the clinic, the systems biology data must be linked to biomedical data, biomedical measurements and personalised patient data. An integration on the syntactical level is not expressive enough to allow for automatisation, but integration on the semantic level holds the promise of overcoming this limitation. Figure 1 illustrates the necessary steps for the semantic integration of patient data, computational models, and external data for the benefit of patients and clinical staff.

Many biomedical ontologies are maintained in online portals, such as BioPortal or the Open Biomedical Ontologies (OBO) Foundry web page, which provide search interfaces, web services, version control, and mappings between ontologies [ 23, 24, 25 ]. However, different ontologies are used for a semantic representation due to e.g. differences in the medical systems used in different countries which requires reliable mappings between these ontologies.

One effort addressing the mapping between terminologies and ontologies is the Unified Medical Language Systems (UMLS) [ 26 ], which to date harmonises over 150 terminologies and ontologies4. For example, the Human Phenotype Ontology [ 27 ], the International Classification of Diseases 5 and SNOMED CT [ 28 ] are all integrated in UMLS. While resources such as UMLS allow the transfer from one ontology to the other, it is important to be aware that this process of transfer largely depends on the quality of the mapping and the quality of the annotations that have been assigned in the first place. Moreover, as ontologies go through several development cycles, the mappings need to be updated, which in itself can lead to a change in the quality of the mapping and consequently the alignment of data and models in clinical applications. Furthermore, research into the direction of mappings and similarity measures for terms within and across bio-ontologies should be taken into account [ 29 ]. For example, it can be valuable to determine the similarity of data sets that are annotated to different ontologies.

Another set of ontologies to consider for this endeavour are those encoding information about model versions, as well as provenance and evidence of data encoded in the model. For example, PROV-O [ 30 ] is an ontology of provenance terms that could potentially be adapted to attach provenance to model data. Similarly, the Provenance, Authoring and Versioning Ontology (PAV) [ 31 ],can be used to add provenance information for collected data and representations chosen in simulation models/modules. Another effort going into this direction is the Ontology of Biomedical AssociatioN (OBAN), used for provenance information on disease-phenotype associations text mined through EuropePMC6 [ 32 ]. Furthermore, the Evidence Ontology (EVO) [ 33 ] captures terms that can be used to trace biomedical evidence in data as well as models. Despite these ongoing efforts, further work is needed to allow for the integration of computational models with a variety of independent data resources.

4 https://www.nlm.nih.gov/pubs/factsheets/umls.

html, accessed 14 June 2016

5 http://www.who.int/classifications/icd/en/

6 http://europepmc.org/

3.2

Generation of safe simulation modules

Reusability depends on the availability of all model-related data [ 8 ]. For studies performed by medical researchers, it is particularly important to provide full documentation of safe parameter ranges and test case scenarios. This requires tailor-made standards for reporting. The data description must ensure that it is straightforward to interpret the output from simulation modules without an expertise in modelling.

In this context, a simulation module encapsulates a computational model that has been tested, documented, annotated, and certified to meet safety requirements. A module suitable for inclusion into a diagnostic tool needs to provide extensive documentation and safe, standardised software interfaces (e. g. for resetting simulation parameters or accessing and interpreting simulation results; see more details section 3.4). The requirements for documentation of a model are clearly defined in a Minimum Information guideline (MIRIAM) [ 34 ]. We argue that the documentation of a simulation module for medical research needs to be extended to also cover information on applicable virtual experiments, allowed applications, and conditions under which the data are applicable in simulations.

In addition to these factors, the development, testing and management of software used for medical purposes will need to follow rules issued by regulatory agencies to ensure the safety of patients and their related data. As medical software Apps have become more prevalent, guidance has been developed by a number of national agencies including Germany (“Medizinproduktegesetz”)7 the US 8 and the UK 9. These include definitions of what software constitutes a “medical device” and which regulations apply. However frameworks to regulate sophisticated software systems for medicine, such as simulation modules, will need considerably more development. 3.3

Testing procedures to ensure safety

Due to the sheer amount of data necessary to model the physiology of a human being, the development of future diagnostic tools will rely on previously developed, standardised simulation modules and on thorough semantic annotation. Before models and consequently modules can be consulted in medical predictions they need to be tested thoroughly. This is, in theory, possible for a subset of models in systems biology. For example, all models in the curated branch of BioModels should be able to reproduce at least one behavior observed and described in the reference publication.

For a module to be considered safe in a clinical environment, the encapsulated model predictions must be medically reliable, i.e. they must not only capture the underlying disease mechanisms but also adapt to the uniqueness of each individual patient. This requirement entails that the error rate for predictions needs to be very small and under no circumstances can exceptions lead to failure in the intermediate computation. Due to the diversity of data that is included into a model, physical units, error ranges and data mappings have to be handled with special care. It is crucial that the patient-specific

7 http://www.bfarm.de/DE/Medizinprodukte/

Abgrenzung/medical_apps/_node.html

8 http://www.fda.gov/downloads/medicaldevices/

deviceregulationandguidance/guidancedocuments/ ucm263366.pdf

9 https://www.gov.uk/government/publications/

medical-devices-software-applications-apps data to be simulated with the module matches the requirements of model parameters such that a reliable prediction can be ensured.

For this purpose, standardised tests need to be in place and continuously be passed throughout development. The electrophysiology web lab [ 35 ] is one example of a web-based tool to check the reliability of models relating to the physiology of the heart. It features a set of published models in CellML format, and applies to them several virtual experiments. The tests check how each model reproduces the expected behavior of a real heart under a variety of conditions. This procedure is referred to as functional curation of the model [ 36 ].

Tests facilitate model evaluation and are thus an important component of a module. The test data consists of simulation inputs and outputs, which allow users to evaluate predictive error, sensitivity and specificity of a module. Furthermore, users require access to the tests with which the parameter ranges and prediction outcomes have been assessed during model development.

The documentation released with a simulation module should detail how simulation results are to be correctly interpreted. This is particularly relevant for the classification of results in terms of quantiles within patient cohorts. In order to verify whether a module is safe for use, information detailing the history, developer(s), input data and test results is strictly necessary. Only if this information is provided one can evaluate if the latest version of a module is safe for application and how the changes made over time have affected the error rates of predictions as well as edge-cases in simulation scenarios. Systems biology already offers tools for model version control (e. g., [ 37 ]). However, we note that the potential of model provenance has not yet been fully explored, and the description of model parameters as well as a model’s quality (in terms of applicability and reliability) is so far neither satisfactory nor standardised. 3.4

Standardised and secure software interfaces

In order to apply modules in clinical practice, standardised software interfaces are required that enable the safe simulation of models (e.g. through restricted parameter ranges), validation of input parameters, support for allometric scaling (of parameters like organ sizes or blood flow), and the evaluation of simulation results in terms of confidence intervals.

It is not unlikely that a model used through a diagnostic tool is administered by a clinician, nurse or other medical staff. The simulation mode must hence include a safe mode in which only defined properties of the model/module can be adapted. However, these defined properties need to cover, at the same time, the uniqueness of each patient so that the simulation can be truly personalised. An adaptation of the above web lab can help to provide clinicians with an overview of possible behaviors of a system given different sets of patient data and clinical investigations.

Software tools such as the Taverna Workflow Suite [ 38 ] or Galaxy [ 39 ] are used for various data analysis tasks in Bioinformatics. Once constructed, the workflows are reusable. Executable protocols can be shared, reused and repurposed. Similarly, high-quality workflows could be provided for standard procedures in the clinic that involve virtual experiments. Tested and trusted workflows can safe clinicians time as they automatise processes that otherwise would require a long time to specialise in.

Moreover, tool and model developers have to safeguard the data that is used as input to the computational model so that patient data cannot be used for other purposes than the treatment of this patient. Otherwise obtaining consent from patients to employ their data for medical purposes will be impossible. There is an arguable potential that the models could be improved over time as the patient data in itself can help tweaking model parameters but this would have to be covered by each patient’s consent. 4

CONCLUSION

With kinetic models being increasingly used and reused for the prediction of disease risks, the monitoring of disease progression, or for drug development, the quality and reliability of models becomes a major concern. In this situation, medical research can benefit from the experiences in systems biology, by incorporating existing standards, tools and infrastructure. Standards and standard-compliant tools increase the exchangeability of models, and enable researchers to reproduce published results. As computational models can be readily parameterised with individual patient and cohort data, they are well-suited for personalisation. Moreover, the models can be embedded in pharmacokinetics and pharmacodynamics applications used during drug development.

However, before modeling can be fully incorporated into medical workflows, additional requirements should be met. Among these are further standards to represent the provenance of a model and to document valid parameter ranges under certain conditions. Furthermore, solutions for high-quality annotation of models and for the curation of data need to be developed. Other challenges, like the representation of uncertainties, restricted model changes and personalisation are yet unsolved and have to be addressed in future research. A specific focus of future works should be on the definition of a minimal semantic interface that patient data has to fulfill for a model to be applicable, i. e., a minimal set of semantically encoded data the model requires as input. For instance, in the case of a regression model, all independent variables of the model must exist.

Finally, models used in the clinic need to fulfill safety requirements and adhere to data privacy guidelines. For example, at no point would it be acceptable to mix data from several patients and give a patient or other unauthorised staff access to patients’ data.

We conclude that systems biology research focuses on the development of (predictive) models. These models are mainly set in a research environment and use batch samples and flexible time tables. Many of the achievements towards reproducibility of simulation studies in systems biology can be reused to establish an infrastructure for reusable models in the clinic. However, the existing infrastructure needs to be evaluated thoroughly, and it needs to be extended to meet clinical standards when working with patient data.

ACKNOWLEDGEMENT

MK is supported by the Federal Ministry of Education and Research (BMBF, Germany) within the research network Systems Medicine of the Liver (LiSyM) (grant number 031L0054). AO and RJBD would like to acknowledge NIHR Biomedical Research Centre for Mental Health, the Biomedical Research Unit for Dementia at the South London, the Maudsley NHS Foundation Trust and Kings College London. RJBD’s work is also supported researchers at the National Institute for Health Research (NIHR) University College London Hospitals Biomedical Research Centre, and by awards establishing the Farr Institute of Health Informatics Research at UCLPartners, from the Medical Research Council, Arthritis Research UK, British Heart Foundation, Cancer Research UK, Chief Scientist Office, Economic and Social Research Council, Engineering and Physical Sciences Research Council, National Institute for Health Research, National Institute for Social Care and Health Research, and Wellcome Trust (grant MR/K006584/1). TJPH would like to acknowledge Kings´ College London and the NIHR Biomedical Research Centre at Guys´ and St ThomasN´ HS Foundation Trust and the NIHR Biomedical Research Centre for Mental Health. DW is funded through the BMBF e:Bio program (grant no. 0316194). The authors acknowledge support through CaSyM, the EC FP7 coordinating action Coordinating Systems Medicine across Europe.

References

[1] Leroy Hood et al. Systems biology and new technologies enable predictive and preventative medicine . Science , 306 ( 5696 ): 640 - 643 , 2004 .

[2] Raimond L Winslow et al. Computational medicine: translating models to clinical care . Science Translational Medicine , 4 ( 158 ): 158rv11 - 158rv11 , 2012 .

[3] Geoffrey S Ginsburg et al. Personalized medicine: revolutionizing drug discovery and patient care . TRENDS in Biotechnology , 19 ( 12 ): 491 - 496 , 2001 .

[4] Olaf Wolkenhauer et al. The road from systems biology to systems medicine . Pediatric research , 73 ( 4-2 ): 502 - 507 , 2013 .

[5] William E Evans et al. Pharmacogenomics: translating functional genomics into rational therapeutics . science , 286 ( 5439 ): 487 - 491 , 1999 .

[6] K Romero et al. The future is now: Model-based clinical trial design for alzheimer's disease . Clinical Pharmacology & Therapeutics , 97 ( 3 ): 210 - 214 , 2015 .

[7] Jessica Nasica-Labouze et al. Amyloid protein and alzheimer's disease: When computer simulations complement experimental studies . Chemical Reviews , 115 ( 9 ): 3518 - 3563 , 2015 .

[8] Dagmar Waltemath et al. How modeling standards, software, and initiatives support reproducibility in systems biology and systems medicine . IEEE Transactions on Biomedical Engineering , June 2016 .

[9] Fiona Cunningham et al. Ensembl 2015 . Nucleic Acids Research , 43 ( D1 ): D662 - D669 , 2015 .

[10] Kate R Rosenbloom et al. Encode data in the ucsc genome browser: year 5 update . Nucleic Acids Research , 41 ( D1 ): D56 - D63 , 2013 .

[11] Taavi Tillmann et al. Systems medicine 2 . 0: potential benefits of combining electronic health care records with systems science models . Journal of Medical Internet Research , 17 ( 3 ):e64, 2015 .

[12] Guido Santos et al. Model-based genotype-phenotype mapping used to investigate gene signatures of immune sensitivity and resistance in melanoma micrometastasis . Scientific Reports , 6 , 2016 .

[13] Jonathan Cooper et al. A call for virtual experiments: accelerating the scientific process . Progress in biophysics and molecular biology , 117 ( 1 ): 99 - 106 , 2015 .

[14] Dagmar Waltemath et al. Reproducible computational biology experiments with sed-ml-the simulation experiment description markup language . BMC systems biology , 5 ( 1 ): 1 , 2011 .

[15] Tommy Yu et al. The physiome model repository 2 . Bioinformatics, 27 ( 5 ): 743 - 744 , 2011 .

[16] Brett G Olivier and Jacky L Snoep . Web-based kinetic modelling using jws online . Bioinformatics , 20 ( 13 ): 2143 - 2144 , 2004 .

[17] Falk Schreiber et al. Specifications of standards in systems and synthetic biology . J. Int. Bioinformatics , 12 ( 258 .10): 2390 , 2015 .

[18] Leonard P Freedman et al. The economics of reproducibility in preclinical research . PLOS Biology , 13 ( 6 ):e1002165, 2015 .

[19] Victoria Uren et al. Semantic annotation for knowledge management: Requirements and a survey of the state of the art . Web Semantics: science, services and agents on the World Wide Web , 4 ( 1 ): 14 - 28 , 2006 .

[20] Chen Li et al. Biomodels database: An enhanced, curated and annotated resource for published quantitative kinetic models . BMC Systems Biology , 4 ( 1 ): 92 , 2010 .

[21] Nick Juty et al. Identifiers. org and MIRIAM Registry: community resources to provide persistent identification . Nucleic Acids Research , 40 ( D1 ): D580 - D586 , 2012 .

[22] John H Gennari et al. Multiple ontologies in action: composite annotations for biosimulation models . Journal of Biomedical Informatics , 44 ( 1 ): 146 - 154 , 2011 .

[23] Manuel Salvadores et al. BioPortal as a dataset of linked biomedical ontologies and terminologies in RDF . Semantic Web , 4 ( 3 ): 277 - 284 , 2013 .

[24] Barry Smith et al. The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration . Nature Biotechnology , 25 ( 11 ): 1251 - 1255 , 2007 .

[25] Anika Gross et al. How do computed ontology mappings evolve?-a case study for life science ontologies . In Joint Workshop on Knowledge Evolution and Ontology Dynamics , 2012 .

[26] Olivier Bodenreider . The unified medical language system (UMLS): integrating biomedical terminology . Nucleic Acids Research , 32 ( suppl 1 ): D267 - D270 , 2004 .

[27] Sebastian Köhler et al. The human phenotype ontology project: linking molecular biology and disease through phenotype data . Nucleic Acids Research , 42 ( D1 ): D966 - D974 , 2014 .

[28] Kevin Donnelly . SNOMED-CT: The advanced terminology and coding system for eHealth . Studies in Health Technology and Informatics , 121 : 279 , 2006 .

[29] Michael Hartung et al. Effective composition of mappings for matching biomedical ontologies . In Extended Semantic Web Conference , pages 176 - 190 . Springer, 2012 .

[30] Timothy Lebo et al. Prov-o: The prov ontology . W3C Recommendation , 30 , 2013 .

[31] Paolo Ciccarese et al. Pav ontology: provenance, authoring and versioning . Journal of biomedical semantics , 4 ( 1 ): 1 , 2013 .

[32] Sirarat Sarntivijai et al. Linking rare and common disease: mapping clinical disease-phenotypes to ontologies in therapeutic target validation . Journal of Biomedical Semantics , 7 ( 8 ), 2016 .

[33] Marcus C Chibucos et al. Standardized description of scientific evidence using the Evidence Ontology (ECO) . Database , 2014 :bau075, 2014 .

[34] Nicolas Le Novère et al. Minimum information requested in the annotation of biochemical models (MIRIAM) . Nature Biotechnology , 23 ( 12 ): 1509 - 1515 , 2005 .

[35] Jonathan Cooper et al. The Cardiac Electrophysiology Web Lab . Biophysical Journal , 110 ( 2 ): 292 - 300 , 2016 .

[36] Jonathan Cooper et al. High-throughput functional curation of cellular electrophysiology models . Progress in biophysics and molecular biology , 107 ( 1 ): 11 - 20 , 2011 .

[37] Martin Scharm et al. An algorithm to detect and communicate the differences in computational models describing biological systems . Bioinformatics, page btv484, 2015 .

[38] Katherine Wolstencroft et al. The taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud . Nucleic acids research , page gkt328 , 2013 .

[39] Jeremy Goecks et al. Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences . Genome biology , 11 ( 8 ): 1 , 2010 .