=Paper=
{{Paper
|id=Vol-2065/paper05
|storemode=property
|title=Improving Publication and Reproducibility of Computational Experiments through Workflow Abstractions
|pdfUrl=https://ceur-ws.org/Vol-2065/paper05.pdf
|volume=Vol-2065
|authors=Yolanda Gil,Daniel Garijo,Margaret Knoblock,Alyssa Deng,Ravali Adusumilli,Varun Ratnakar,Parag Mallick
|dblpUrl=https://dblp.org/rec/conf/kcap/GilGKDARM17
}}
==Improving Publication and Reproducibility of Computational Experiments through Workflow Abstractions==
https://ceur-ws.org/Vol-2065/paper05.pdf
Improving Publication and Reproducibility of Computational
Experiments through Workflow Abstractions
Yolanda Gil Daniel Garijo Margaret Knoblock
Information Sciences Institute Information Sciences Institute Information Sciences Institute
University of Southern California University of Southern California University of Southern California
USA USA USA
gil@isi.edu dgarijo@isi.edu mrk022@bucknell.edu
Alyssa Deng Ravali Adusumilli Varun Ratnakar
Information Sciences Institute School of Medicine Information Sciences Institute
University of Southern California Stanford University University of Southern California
USA USA USA
shipingd@andrew.cmu.edu ravali@stanford.edu varunr@isi.edu
Parag Mallick
School of Medicine
Stanford University
USA
paragm@stanford.edu
ABSTRACT 1 INTRODUCTION
The current practice of publishing articles solely containing The reproducibility crisis in science has received
textual descriptions of methods is error prone and incomplete.
significant attention. Reproducibility requires that methods
Even when a reproducible workflow or notebook is linked to an
article, the text of the article is not well integrated with those
are described with enough details to repeat the experiment
computational components, and the workflow and notebook are in an independent lab or setting. For computational
focused mostly on implementation details that are disconnected experiments, studies show that published papers often
from the scientific approach described in the text of the article. provide insufficient information about the data, protocols,
Through an analysis of three multi-omics articles, we illustrate software, and overall method used to obtain the new results
why this makes it difficult to understand, reproduce, compare, and [Van Noorden 2015]. A major barrier to reproducibility
reuse computational methods. We propose workflow abstractions can be traced to the traditional, unstructured format of
that that capture different concepts and perspectives that are publications “materials and methods” sections. The
important to scientists. These abstractions connect the text of an
ambiguity, imprecision, and linearity of text make natural
article to the corresponding workflow, and provide a framework
to improve the publication and reproducibility of computational
language descriptions of computational analyses inadequate
experiments. for reproducible research [Steehouder et al 2000; Garijo et
al 2013; Gil 2015; Groth and Gil 2009]. A major problem
CCS CONCEPTS is that there is no guidance or methodology to describe
• Information systems → Artificial intelligence; Knowledge computational methods in articles. It is unclear what the
representation and reasoning intent of the descriptions in methods sections is. Is the goal
to provide a step-by-step account of the procedures taken,
KEYWORDS parameters employed, and data provenance such that a
Reproducibility, semantic workflows, semantic science study might be reproduced? Alternately, is the goal to
provide a high-level intuition for the steps that were
performed? Both are valuable but are incompatible
objectives in current text-based descriptions, leading to
K-CAP2017 Workshops and Tutorials Proceedings,
neither an intuitive reading experience nor a reproducible
© Copyright held by the owner/author(s) description.
K-CAP’17 SciKnow, December 2017, Austin, TX, USA Y. Gil et al.
Computational workflows and notebooks can be used to from the text of the paper how the work was done. The
organize and record computational methods, and are often reproducibility maps showed that only researchers with the
linked to publications. Workflows capture the dataflow same level of expertise in the subject as the authors were
among computations, so the different steps of the method able to figure out how to fully reproduce the work. There
are explicitly represented and linked. Notebooks are are many similar results in the literature, some mentioning
composed of cells that contain either text or code that can the lack of publication of data [Ioannidis et al 2009] and
be easily re-run. Workflows and notebooks facilitate the others the lack of details in the description of methods
documentation of the software and the structure of a leading to “exercises in ‘forensic bioinformatics’ where
computational method. But even when workflows and aspects of raw data and reported results are used to infer
notebooks are used, the text for those publications is always what methods must have been employed” [Baggerly and
manually generated by the authors and often inadequately Coombes 2009]. There are several reasons why text
captures the full complexity of an analysis, leading to poor descriptions of methods are riddled with problems. First,
reproducibility. In prior work, we developed an approach articles often have space limitations, so authors tend to omit
to automatically create descriptions of computational anything that seems not important. Second, they are
methods by generating text from workflows [Gil and Garijo manually written without any particular guidance, it is easy
2017], where the text accurately represents what was done for authors to provide imprecise descriptions. Finally,
and can be presented from different perspectives. The text, computational methods are often complex procedures with
however, can only be as good as the workflows that it was non-linear structures that are hard to describe with the
generated from. This motivates the need for a workflow sequential nature of text [Gil 2015]. Even when authors
design methodology that leads to workflow representations endeavor to describe enough details, textual descriptions
that support the automated generation of explanations as are often ambiguous. A study reported in [Ince et al 2012]
text that can be used in publications. looked at writing software from scratch based on the
This paper proposes abstractions designed to improve textual descriptions reported in geophysics papers and
the publication of computational methods to facilitate found radical differences in the implementations. The
reproducibility. This methodology extends our prior work papers were found to be ambiguous at the lexical, syntactic,
on representing and publishing workflow abstractions using and semantic level, and not necessarily because the authors
community standards for workflows and provenance were not rigorous but because natural language is
[Garijo et al 2017]. inherently ambiguous. We also find that the methods
The paper begins with an overview of related work on sections of articles mix general methods with specific
reproducibility and publication of computational methods. details of the executions carried out [Gil and Garijo 2017].
We then present an analysis of the computational methods Although there are many tools and recommendations of
described in three seminal papers in cancer omics. We best practices for authors [Stodden et al 2016], it is still up
introduce the abstractions proposed, and discuss their to them to figure out what to include in an article and its
merits in improving the descriptions of computational methods section. In summary, textual descriptions of
methods in scientific publications. methods in articles are far from ideal, since the text tends to
be: 1) Incomplete, omitting important details about the
2 PUBLICATION AND computations performed; 2) Ambiguous, having several
REPRODUCIBILITY OF interpretations of how the computations were actually
COMPUTATIONAL METHODS done; 3) Mingled, interspersing general overviews with
execution details.
Textual descriptions of methods in articles may be
Workflows capture unambiguously a computational
incomplete (e.g., [Ioannidis et al 2009; Donoho et al
analysis as a dataflow among steps [Taylor et al 2006]. In
2009]). Authors focus on conveying the major
prior work, we found that workflow reusability is a major
contributions of the work and describe the methods in that
drive for users [Garijo et al 2014a]. Workflow repositories
light, omitting details that may be important for
provide mechanisms to publish and search workflows,
transparency and reproducibility. For example, [Garijo et al
particularly to improve reproducibility and sharing of
2013] describes our work to reproduce an article for which
computational experiments. However, the descriptions of
the authors had provided the data, software, and results to
workflows are manually generated and therefore are as
facilitate reproducibility. We created reproducibility maps,
incomplete as those in scientific articles. In prior work we
that showed different categories of users could figure out
2
A Workflow Design Methodology to Improve Reproducibility K-CAP’17 SciKnow, December 2017, Austin, TX, USA
analyzed the textual descriptions of workflows from one of [TCGA 2008] and [Imielinski et al 2012] which describe
these repositories [Groth and Gil 2009]. We found work on genomics that Zhang and colleagues built upon. A
significant differences between what was included in the detailed analysis of all three articles is provided in
textual descriptions and the actual formal specification of [Knoblock 2017].
the workflows. A major limitation of workflow
representations is that they mix major method steps with 3.1 Method Descriptions
ancillary steps that do for example minor data reformatting.
The methods section of a scientific article describes,
Also in previous work, we analyzed workflows to identify
together with the supplementary materials, the data and
by hand general categories of steps (motifs) that make such
computational steps used for data analysis. We illustrate
distinctions [Garijo et al 2014b]. But workflows in
how methods are typically described using excerpts from
themselves have no explicit mention of the relative
[TCGA 2008] for variant calling from resequencing data,
importance of steps and all steps are treated equally. In
where SNPs and indels were screened against dbSNP for
summary, although workflows provide a formal
position/allele match. First, “Putative variants were
computational representation of methods, the workflows
identified using Polyphred 6.1, Polyscan 3.0, SNPdetector
themselves are: 1) Incomplete, because workflow
3, and SNP Compare. SNPs and indels were screened
representations do not express important semantic
against dbSNP for position/allele match”. This excerpt
properties of steps; 2) Flat, with abstractions often absent
describes the software, although it refers to entire packages
from the workflow structure; and 3) Undifferentiated, as
and not how they are used in the method. Then,
there is no explicit distinction between important steps and
“Boundaries of insertion, deletion and complex
ancillary steps.
rearrangements [were] annotated”, and the detailed
A recent popular trend is electronic notebooks, such as
annotation guidelines are outlined in the article. That
Jupyter Notebook and Apache Zeppelin, where the
excerpt focuses on the science method. Next, “The first
advantage is that the text is intermixed with data and code
step in analysis of the mutation data was to combine the
so it is easier to follow step by step how the method
.maf files from all centers into a .mut file containing at
actually works. This approach is akin to executable papers
most one record for each site-sample pair. In the process of
which have been around for some time, such as Sweave
combining the files, care was taken to detect and resolve
and knitr which combine Latex and R [Xie 2015].
conflicts between multiple records for the same site-
However, a reader cannot easily compare two notebooks,
sample”. This excerpt is focused on low level
since that requires comparing the code line by line, and
implementation aspects such as file formats and handling
cannot easily reuse parts of one to create another since the
duplicate entries. And finally, “As part of our sequencing
code in notebook cells is not necessarily modular. In
pipeline, non-synonymous mutations were subjected to an
addition, although notebooks are easily published and
orthogonal validation or re-sequencing (verification) step
shared they have not replaced published papers, possibly
to decrease the prevalence of false positives. In our
due to their idiosyncratic formats which do not yet offer the
analysis we considered only those mutations that were
persistence and archival guarantees required by publishers.
confirmed by validation or verification to be actual somatic
In summary, in order to understand a published article,
mutations”. This excerpt focuses on the science aspects of
and assess its validity, reproduce the work, or to compare
the analysis, but it does not specify how each step of the
its method to another article, a reader must do a significant
method are implemented by the software packages
amount of work. Even when authors capture computational
mentioned earlier.
methods as workflows, there is no guidance on how to
This is a common approach to describing methods.
facilitate reproducibility and reuse. The next section
Method descriptions in scientific articles mix mentions
analyzes specific articles in detail as motivating scenarios,
of software, data formats, and scientific descriptions of
and extracts desiderata for workflow design.
the experiment.
Figure 1 shows a workflow that a biologist created
3 AN ANALYSIS OF MULTI-OMICS based on the method description in the article. Not
METHODS surprisingly, the workflow steps are also mixture of science
This section motivates our work in the context of three concepts and software implementation and data formats.
seminal articles in multi-omics: [Zhang et al 2014], which Note that this makes it hard for another scientist to
is the first publication of a large-scale multi-omics analysis, understand and therefore reuse the workflow.
3
K-CAP’17 SciKnow, December 2017, Austin, TX, USA Y. Gil et al.
Base caller (Phred software): Assembling genome
for genomic alignment/features.
Genome assembly (Consed software): Calling
genomic bases from input files.
SNP caller (Polyphred software): Calling SNPs
from the input genomic file (variant calling).
Indel and SNP caller: (PolyScan software):
Calling both indels and SNPs (variant calling).
Variant annotation (Annotate_ MAFFormat
software): Annotating variants based on reference
genomes.
Join data files (mutipleFilesToOne software):
Appending multiple text files.
Filter data files (filt_MAF_file software): Variant
filtering based on input parameters.
We make a few observations about how the software is
presented in the article and used in the computational
method.
Paper descriptions of conceptual steps contain very
limited information about how they map to software.
Given the capabilities of the software used, conceptual
steps may be mapped to several implemented steps, and
vice versa. For example, Polyphred and Polyscan are two
separate steps, both implement variant calling but the
former does SNP calling only and the latter implements
indel and SNP calling steps. Therefore, a requirement in
designing a workflow is that it must make clear how each
step is implemented in software.
Paper descriptions of software contain limited
information about what conceptual steps they
Figure 1: Computational workflow to annotate variant implement. For example [TCGA 2008] says: “Putative
calling for resequencing data, based on [TCGA 2008]. variants were identified using Polyphred 6.1, Polyscan 3.0,
Each workflow step (square boxes) is described as the SNPdetector 3, and SNP Compare.” Four pieces of
software that implements the step. software are mentioned, but there are no details that specify
what types of variants are detected by each of them.
A scientist may not be familiar with the different Therefore, in designing a workflow, the mapping of
software packages (there are hundreds of packages that are conceptual steps to software must be clearly stated.
available for this kind of analysis), and therefore would not A given function can be implemented by many
understand the function of each step. Therefore, the design software packages. There are many software packages that
of workflows should accommodate the separation between provide a desired functionality. As a result, identical
the conceptual description of the experiment and the functions in different methods may be implemented by
implementation of the experiment in software. different software, making it hard for a scientist to compare
workflows. For example, in the workflow in Figure 1 the
Consed software is used to perform the genome assembly
3.2 Software Descriptions
step. In [Zhang et al 2014], Tophat2 performs this genome
Each step in a method can be described at a conceptual assembly step. Therefore, a requirement is that the software
level in terms of the function that it performs, and at an steps be described according to their functionality, so that
implementation level as the software used for the step. For the methods for several papers can be more easily
the articles we analyzed, these descriptions are as follows: compared by a scientist. Functions should be specified for
4
A Workflow Design Methodology to Improve Reproducibility K-CAP’17 SciKnow, December 2017, Austin, TX, USA
each method step so that the correspondences across workflow is to represent explicitly what formats are
different software implementations for the same conceptual imposed by the use of specific software packages.
step will be explicit. Data of the same type can play very different roles
A given software package has many functions. In in a method. In the workflow in Figure 1, the
comparing software to the workflows built from them, we varAnnotParams input is an annotated variant parameter
found that many scientific software packages have a large file but this type is not represented explicitly. Moreover, it
number of functions. Though it is useful for scientists to is also not the only annotated file that is input to this
have multiple functions in one software package, in method, but is the only one with a name that mentions
research papers it can be difficult to tell what software annotation. Therefore, a requirement is to describe data
packages are being used for what functions. Sometimes the conceptually according to the type of data contained, and
functionality of a software package is quite broad. For that different data used or generated in the workflow be
example, the SAMtools software package, used in [Zhang related by those types.
et al 2014], can be used for Variant Calling and Variant Data results of the same type may be combined,
Filtering but the article does not explicitly indicate for what filtered, or sorted in ways that are not considered
function it is used. Therefore, when specifying what important to mention in the paper. Readers must
software is used to implement a step in a method, it is vital hypothesize these data manipulations.
to indicate the specific function of that software to make it
unambiguous what conceptual function the software is 3.4 Discussion
implementing.
Through examples we have illustrated that the text
A computational step may perform a data
descriptions of methods sections of articles makes them
reformatting, conversion, or other minor step that is not
hard to reconstruct and replicate into an unambiguous and
conceptually important and therefore is not mentioned
complete workflow. This is because papers describe
in the article. Without a description of these steps, it may
methods in a mix of high-level conceptual terms together
not be possible to interpret the results appropriately or to
with mentions of specific software and formats. This
reproduce the method.
makes it hard to understand and compare methods.
In summary, the descriptions of method steps and their
Another observation is that different readers might be
implementation in software that are typically found in
interested in different descriptions of the methods, some
scientific articles are very ambiguous and incomplete.
more abstract and some more specific. For example, a
Computational workflows can eliminate this ambiguity, but
developer would be interested in data formats and software
they must be intentionally designed to be unambiguous and
versions, while a biologist would be more interested in the
complete.
overall statistical approach used.
Ideally, method descriptions would make clear
3.3 Data Descriptions distinctions between high-level conceptual terminology and
Like software, data is described in scientific papers with a implementation terms, both for software and for data. In
mixture of high-level concepts and low-level format addition, method descriptions would make it clear what
references. function each step performs, and whether a given function
Data is often described based on its format rather is implemented by a single step or by a set of steps. These
than its contents. We saw examples of this in the earlier desiderata lead us to propose workflow abstractions for
article excerpts. Therefore, a requirement in the design of describing computational experiments in a paper.
workflows is that data abstractions should be used to
complement step abstractions. 4 WORKFLOW ABSTRACTIONS
Data formats are sometimes used when data is A computational method is typically described in terms of
generated in idiosyncratic formats by specific software the specific software, data, and formats used. However,
used. This can be seen in Figure 1. The Phred software there are many ways to describe a method
generates output in a format called phd, and as a result the conceptually. This section describes different ways to
workflow indicates phd_File which is specific to Phred. design workflow abstractions that would be useful to make
Thus, a user of the workflow unfamiliar with Phred would methods more understandable and comparable. Table 1
find it hard to understand that format. Therefore, a summarizes the issues identified earlier and the
requirement in describing software steps and data in a corresponding proposed abstractions to address them.
5
K-CAP’17 SciKnow, December 2017, Austin, TX, USA Y. Gil et al.
Table 1. An overview of the issues identified in the papers
analyzed and abstractions proposed to address them.
Issue identified Abstraction
approach
1) Method descriptions in scientific articles Step
mix mentions of software, data formats, and abstractions
scientific descriptions of the experiment
2) Paper descriptions of conceptual steps Sub-
contain very limited information about how workflow
they map to software and step
abstractions
3) Paper descriptions of software contain Sub-
limited information about what conceptual workflow
steps they implement and step
abstractions
4) A given function can be implemented by Step
many software packages abstractions
5) A given software package has many Step
functions abstractions
6) A computational step may perform a data Criticality
reformatting, conversion, or other minor abstractions
step that is not conceptually important
7) Data is often described based on its Data
format rather than its contents abstractions
8) Data formats are sometimes used when Data
data is generated in idiosyncratic formats by abstractions
the specific software used Figure 2: A computational workflow that corresponds to
9) Data of the same type can play very Data the workflow in Figure 1 but where each step is described
different roles in a method abstractions conceptually.
10) Data results of the same type may be Criticality
combined, filtered, or sorted in ways that abstractions At the same time, the software steps in Figure 1 and the
are not considered important to mention in conceptual steps of Figure 2 should be mapped to one
the paper another. This can be done through a hierarchy of
component functions, which defines many conceptual
functions at different levels of detail. The hierarchy
4.1 Step Abstractions bottoms out with mentions of software that implements the
A computational workflow can be described at a conceptual parent function. Note that there may be several software
level in terms of the functions that each step carries out. implementations of the same abstract function.
Figure 2 describes the same workflow introduced in Figure
1. While Figure 1 describes the software implementation
of each step, Figure 2 characterizes the function of each
step.
6
A Workflow Design Methodology to Improve Reproducibility K-CAP’17 SciKnow, December 2017, Austin, TX, USA
very helpful to a reader. Workflow designers should design
appropriate conceptual levels.
A hierarchy of component functions becomes a
powerful enabler for automation. Given a concrete
workflow, the hierarchy could be used to generate abstract
workflows automatically. Conversely, given an abstract
workflow, the hierarchy could be used to specialize it and
create a concrete workflow. [Gil et al 2011] describe
algorithms to do this kind of automation.
4.2 Sub-Workflows
Several components may implement different aspects of the
same function. For example, in the workflow of Figures 1
and 2 the Polyphred software and the Polyscan software
implement SNP calling and indel calling respectively,
Figure 3: A hierarchy of component functions to describe which are two aspects of variant calling. The software
method steps. The software steps in Figure 1 are shown in Annotate_MafFormat annotates the resulting variants with
dark blue, and the abstract steps in Figure 2 are shown in respect to reference genomes. All three steps could be
green, both from [TCGA 2008]. Additional steps in considered as a sub-workflow, with an overarching abstract
[Imielinski et al 2012] are shown in purple, and those in function of detecting and annotating variants.
[Zhang et al 2014] are shown in light blue. A knowledge base of sub-workflows would capture these
functional decompositions. A sub-workflow would consist
Figure 3 shows a hierarchy of component functions for of a root abstract component, which indicates the
the steps in Figures 1 and 2. The function in a given node overarching abstract function, and a workflow fragment that
represents a more specific function than the function of its decomposes that function into a set of components at a
parent node. It also includes steps for the other two articles lower level of abstraction and the dataflow among them.
that we analyzed. Using this hierarchy, it becomes possible Data abstractions should be taken into account as well as
to relate the method steps of the three articles. the sub-workflows express functions of different
When designing a workflow, two distinct types of abstraction levels. We discuss data abstractions below.
workflows should be created. One type of workflow is an When designing a workflow, steps that are functionally
abstract workflow, with abstract components that related should be organized as sub-workflows. There may
correspond to the more general functions in the hierarchy. be alternative ways to group steps in a workflow.
These abstract workflows capture the general functionality Workflow designers should make decisions based on the
of methods, and they would be independent of the software expected use of the sub-workflow decompositions by
used to implement it. A second type of workflow would be readers. The knowledge base of sub-workflows could be
a grounded workflow, which would specify what software dynamically extended based on a growing corpus of
is used to implement each step. workflows created by users. [Garijo et al 2014c] describe
We find that in practice it is hard to create a complete techniques to detect workflow fragments automatically.
hierarchy of component functions before creating the
workflows. We recommend an iterative process, where an 4.3 Criticality
initial hierarchy is created and then refined as the Some steps in a workflow perform functions that are
workflows are fleshed out. critical to the overall computational method, while other
Depending on the depth of the hierarchy of component steps carry out minor format conversions and other
functions there could be several abstract workflows that ancillary functions. For example, the workflow in Figures
could have different levels of detail and generality. Each 1 and 2 has a step to merge several files. Other workflows
abstract workflow may be useful to a different reader, have reformatting steps, unit conversion steps, and other
depending on the level of detail that they are looking to functions that manage the details of how the data is
find. At the same time, if the workflow contains implemented. When describing a method in a paper, these
descriptions of the steps that are too general, it may not be ancillary functions are rarely mentioned. There may be
7
K-CAP’17 SciKnow, December 2017, Austin, TX, USA Y. Gil et al.
different degrees of criticality, depending on how much 5 CONCLUSIONS
detail each reader is interested in seeing.
This paper motivates the need for capturing abstractions in
This kind of abstraction could be captured in a hierarchy
the design scientific workflows. These abstractions are
of criticality levels. This hierarchy would identify the
based on our analysis of published articles and the
importance of including a step in a scientific description of
workflows created to reconstruct their methods. The
a method. [Garijo et al 2014b] describe an approach to
proposed abstractions are captured in hierarchies of
identifying criticality based on a library of workflow motifs
component functions and criticality as well as knowledge
that include data pre-processing, visualization, and format
bases of sub-workflows, and need to be supported by data
conversion. Criticality levels are highly dependent on the
abstractions. Using these abstractions, different workflows
specific domain, but a broad methodology to design those
can be created to describe the same computation for readers
categories could be more generally designed.
with different interests. In future work, we plan to develop
these abstractions for a target domain and associated
4.4 Data Abstractions publications, in order to demonstrate their benefits.
Data type abstractions should be included in all three
hierarchies above. The data type in a node would represent Acknowledgements. We gratefully acknowledge support
data that is of a more specific type than its parent node, for from the Defense Advanced Research Projects Agency
example because it is of a subtype or has more specific through the SIMPLEX program with award W911NF-15-1-
metadata properties. In the hierarchy of component 0555, the National Institutes of Health with awards
functions, each abstract component function should specify 1U01CA196387 and 1R01GM117097, and the Canary
inputs and outputs in terms of those general types. At the Foundation.
bottom of the hierarchy, a component is specified with a
specific software invocation, including the exact command REFERENCES
line call to invoke the software and all the input data types [Baggerly and Coombes 2009] Baggerly KA, and KR
and formats that the software expects. In the hierarchy of Coombes. “Deriving Chemosensitivity from Cell Lines:
sub-workflows, the root component may refer to data types Forensic Bioinformatics and Reproducible Research in
that are more abstract than those of the workflow fragment. High-Throughput Biology.” Annals of Applied Statistics 3
Data abstractions depend on the domain. In multi- (4), 2009.
omics, there are many aspects of data that can be described [Donoho et al 2009] Donoho DL, Maleki A, Rahman IU,
in very specific terms but can be abstracted away when Shahram M, and V Stodden. “Reproducible Research in
describing an experiment in scientific terms. Computational Harmonic Analysis.” Computing in Science
Characteristics of a dataset that can lead to useful data & Engineering 11 (1): 8–18, 2009.
abstractions include: 1) type of sequence, such as RNA,
[Garijo et al 2013] Garijo D, Kinnings S, Xie L, Xie L,
DNA, etc.; 2) annotations on those sequences, such as
Zhang Y, Bourne PE, and Y Gil. “Quantifying
indels, CNVs, SNPs, etc.; 3) formats that are often imposed
Reproducibility in Computational Biology: The Case of the
by how software works, such as FASTA, MAF, phd, etc.;
Tuberculosis Drugome.” PLoS ONE 8 (11), 2013.
4) level of detail or accuracy on the sequences, for example
sequences obtained with next-generation sequencing [Garijo et al 2014a] Garijo D, Corcho O, Gil Y, Braskie
machines are more accurate; 5) the role of a dataset for a MN, Hibar D, Hua X, Jahanshad N, Thompson P, and Toga
specific component, for example a sequence can be a AW. “Workflow Reuse in Practice: A Study of
patient sequence or a reference sequence. Neuroimaging Pipeline Users.” Proceedings of the 10th
Workflow designers should create a taxonomy of data IEEE International Conference on e-Science, 2014.
abstractions that facilitate the abstractions needed for the [Garijo et al 2014b] Garijo D, Alper P, Belhajjame K,
three hierarchies discussed earlier. In our work, we have Corcho O, Gil Y, and C Goble. “Common Motifs in
found that a proliferation of data types makes the creation Scientific Workflows: An Empirical Analysis.” Future
of workflows more complex. Instead, we create properties Generation Computer Systems 36, 2014.
for describing the different characteristics of data. [Garijo et al 2014c] Garijo D, Corcho O, Gil Y, Gutman
BA, Dinov ID, Thompson P, and AW Toga. 2014.
“FragFlow: Automated Fragment Detection in Scientific
8
A Workflow Design Methodology to Improve Reproducibility K-CAP’17 SciKnow, December 2017, Austin, TX, USA
Workflows.” Proceedings of the 10th IEEE International Annual Conference on Computer Documentation
Conference on e-Science, 2014. (IPCC/SIGDOC 2000)
[Garijo et al 2017] Garijo D, Gil Y, and O Corcho. [Taylor et al 2006] Taylor IJ, Deelman E, Gannon DB, and
“Abstract, Link, Publish, Exploit: An End to End M Shields. “Workflows for e-Science: scientific
Framework for Workflow Sharing.” Future Generation workflows for grids.” Springer, 2006.
Computer Systems, 2017. [Van Noorden 2015] Van Noorden, R. Sluggish data
[Gil 2015] Gil, Y. “Human Tutorial Instruction in the sharing hampers reproducibility effort. Nature, 2015.
Raw.” ACM Transactions on Interactive Intelligent [Xie 2015] Y Xie. “Dynamic Documents with R and
Systems, 5 (1): 1–29, 2015. knitr.” CRC Press, 2015.
[Gil and Garijo 2017] Gil Y, and D Garijo. “Towards [Zhang et al 2014] Zhang B, Wang J, Wang X, et al.
Automating Data Narratives.” Proceedings of the ACM “Proteogenomic Characterization of Human Colon and
Conf. on Intelligent User Interfaces, 2017. Rectal Cancer.” Nature 513 (7518): 382–87, 2014.
[Gil et al 2011] Gil Y, Gonzalez-Calero PA, Kim J, Moody
J, and V. Ratnakar. “A Semantic Framework for Automatic
Generation of Computational Workflows Using Distributed
Data and Component Catalogs.” Journal of Experimental
and Theoretical Artificial Intelligence, 23(4), 2011.
[Groth and Gil 2009] Groth P and Y Gil. “Analyzing the
Gap between Workflows and Their Natural Language
Descriptions.” Proceedings of the IEEE International
Workshop on Scientific Workflows (SWF), 2009.
[Knoblock 2017] Knoblock M. “Designing Useful
Abstractions for Multi-Omics Data Analysis.” Technical
Report, Information Sciences Institute, University of
Southern California, October 2017.
[Imielinski et al 2012] Imielinski M, Berger AH,
Hammerman PS, Hernandez B, et al. “Mapping the
hallmarks of lung adenocarcinoma with massively parallel
sequencing.” Cell;150(6):1107-20, 2012.
[Ince et al 2012] Ince DC, Hatton L, and J Graham-
Cumming. “The Case for Open Computer Programs.”
Nature, Vol 482, 2012.
[Ioannidis et al 2009] Ioannidis JPA, Allison DB, et al.
“Repeatability of Published Microarray Gene Expression
Analyses.” Nature Genetics 41 (2), 2009.
[TCGA 2008] The Cancer Genome Atlas (TCGA)
collaboration. “Comprehensive Genomic Characterization
Defines Human Glioblastoma Genes and Core Pathways”.
Nature, 455, 1061-1068, 23 October 2008.
[Stodden et al 2016] Stodden V, McNutt M, Bailey DH,
Deelman E, Gil Y, Hanson B, Heroux MA, Ioannidis JP,
and M Taufer. “Enhancing Reproducibility for
Computational Methods.” Science, 354, 2016.
[Steehouder et al 2000] Steehouder, M., Karreman, J. and
Ummelen, N. Making sense of step-by-step procedures.
Proceedings of 2000 Joint IEEE International and 18th
9