-Data processing in data intensive scientific fields like bioinformatics is automated to a great extent. Among others, automation is achieved with workflow engines that execute an explicitly stated sequence of computations. Scientists can use these workflows through science gateways or they develop them by their own. In both cases they may have to preprocess their raw data and also may want to further process the workflow output. The scientist has to take care about provenance of the whole data processing pipeline. This is not a trivial task due to the diverse set of computational tools and environments used during the transformation of raw data to the final results. Thus we created a metadata schema to provide provenance for data processing pipelines and implemented a tool that creates this metadata during the execution of typical scientific computations.
Originally the term provenance is widely used in the field
of arts to ensure the passage of paintings through different
owners. Other fields like archeology also uses provenance
information to exactly describe the find spot of artifacts. In
an analogous way the term data provenance in computational
science describes the lineage of data [
Proper recording of provenance information has many
advantages. One important point is the reliability and
reproducibility of published results. If there is no or just weak
provenance for the result, it is not really reliable [
In this work we present our tool Dataprov for automatic generation of provenance metadata. It was our goal to develop a tool that can record provenance data for arbitrary command line tools and workflow management systems. A researcher can use our tool to track provenance for computations on resources the researcher has access to. At the same time a provider of ready-to-use scientific pipelines can utilize our tool to create provenance metadata for pipeline runs.
The novelty of Dataprov is its extensibility and the support for tracking metadata of used software containers. If a command uses a Docker or Singularity container it is automatically inferred and metadata of the container is incorporated into the resulting provenance file.
The recorded provenance data should be readable for human beings as well as machines. Dataprov should be convenient to use without huge changes to existing scripts or workflows.
In Section II we focus on the provenance data format and
discuss which questions we want to answer with the recorded
provenance data and how we achieve that. Then we describe
how the provenance data is recorded with our tool Dataprov
for arbitrary command line tools as well as for Snakemake
workflows [
Important information that should be provided by
provenance data are already stated in various publications [
Who computed this result? How was the result computed? What input files were used and what is the provenance chain of the input? What was the environment on which the computation was executed?
Was the input or resulting data changed in the meantime? We chose to represent the provenance data in the XML format. The XML format has the advantage that humans as well as machines can read them and that any produced XML document can be validated against a XML schema. The XML schema files for our provenance data format is available in the Dataprov GitHub repository 1.
Figure 1 shows the first three levels of the XML schema. A data provenance (Dataprov) object provides metadata for a file (target). We store the URI of the file and the SHA1 hashsum. The checksum can be used to detect changes since file creation. The history contains a list of operations that lead to the described file.
The XML schema of such an operation is depicted in Figure 1. An operation can have several input files and can produce several output files. As shown in Figure 1 a file is characterized by the triple of filename, URI, and a SHA1 hashsum. The operation is performed by a person (executor) on a specific machine (host). The opClass element describes the operation in more detail and one can provide a message in which the purpose of the operation can be noted.
The operation class (opClass) element is the extensible part
of the schema. Currently our implementation supports shell
commands, commands executed in Docker [
For shell commands just the executed tool and its version is of importance. In contrast, for Snakemake workflows also the workflow file, an optional configuration file, and several workflow steps have to be tracked.
Listing 1. XML Element describing the executor of an operation. <e x e c u t o r > < t i t l e >Dr . </ t i t l e > <f i r s t N a m e >John </ f i r s t N a m e > <surname>Doe</surname> <mail>j o h n . doe@any u n i . edu </ mail> < a f f i l i a t i o n >Group f o r Awsome Things </ a f f i l i a t i o n > </ e x e c u t o r >
The implementation of the data provenance schema in Section II was done with Python and was tested with Python 2.7 and Python 3.6. The tool called Dataprov is available from GitHub2. The general idea is to wrap computations with our tool in the following way: dataprov <options> run <wrapped_command>
Listing 2. XML Element describing the host machine of an operation. <h o s t> <system>Linux </ system> <d i s t >CentOS Linux </ d i s t > <v e r s i o n >7.3.1611 </ v e r s i o n > <codename>Core </codename> <k e r n e l V e r s i o n >4.9.34 29. e l 7 . x86 64 </ k e r n e l V e r s i o n > <machine>x86 64 </machine> <p r o c e s s o r >x86 64 </ p r o c e s s o r > <hostname>a n y h o s t n a m e . any u n i . edu </hostname> </ h o s t>
The operation class element of the XML schema was introduced in Section II. Based on the first part of the wrapped command Dataprov decides which specific element should be generated. E.g. if the wrapped command starts with ’snakemake’, a snakemake workflow element will be generated.
The simplest mode of Dataprov wraps shell commands. The user has to define input files and expected output files manually. One example would be dataprov -i FILE_IN -o FILE_OUT run COMMAND # Example dataprov -i shakespeare.txt -o wc.txt run \ ’wc shakespeare.txt > wc.txt’
The tool checks if provenance files for the input files are available. If yes, the history of the input provenance data is incorporated into the resulting history. Details about the host architecture, running operation system are determined using the Python package platform. We show an example of the collected data in Listing 2. Information about the executor are stored in a configuration file and provides among others Listing 3. XML Element describing a simple command line operation with one input and one output file. elements.
The fact that every schema element has a corresponding counterpart in the Python code makes it convenient to support new tools or workflow systems. In the first step the new XML element for the new operation class is written. In the second step on creates the corresponding Python class. This class contains functions interacting with the wrapped command in order to fill the XML schema with meaningful information. name and affiliation as showed in Listing 1. When Dataprov is first used this configuration file is created in the home directory and the user is asked to fill in the needed information. When the user wraps an operation with Dataprov the executor information from the configuration file is incorporated into the resulting provenance metadata.
For simple command line options we try to infer the location of the executable with the find executable function of the Python distutils package and the software version by running a subprocess of the tool with both the –version/-v switches and capture the output.
The wrapped commands are executed as subprocesses if possible. The output of the subprocess is also written to the shell such that no important output is masked. After the subprocess has finished, we write the collected provenance data for each output file to an XML file. If an error occurrs during the subprocess no provenance data is written.
In the word count example above one has to specify the input and output files twice. First for Dataprov, the other time in the wrapped command. This is tedious and also error-prone, but for a simple wrapper without any alternative.
Specifying the input and output files twice is dropped if one implements support for specific commands into Dataprov. An example for this is the implemented CWL command line tool. To use the CWL reference implementation cwltool one has to specifiy an cwl-file and an input binding file. Dataprov infers information about input and output files by using methods provided by the cwltool Python package. Then the wrapped command will be executed. dataprov run cwltool CWL_FILE INPUT_BINDING
For each element of the XML schema there is a corresponding class in the source code. The class implements basic functionalities like creating Python objects from XML trees and vice versa. Each element is described in its own schema file, whose path is also set in the corresponding Python class. This greatly simplifies the validation of read or written XML Based on three use cases we show how the provenance data is generated. The first use case comprises individual computations. These play a decisive role during the development of new workflows when a researcher tries several tools for a specific task. With good provenance the researcher can track which tool and parameter combination yielded a specific result. It plays also a role for individual computations whose results are used by workflows as input data like indices for reference genomes . The second use shows how we handle provenance information for Docker containers whereas the third use case handles whole scientific workflows.
The researcher tries various tools and parameter choices to evaluate what fits best for the workflow. Usually this is done on the command line in an interactive manner and the result files are stored somehow in a directory hierarchy. Hence, one has to keep track from which tool and parameter combination a result file originates. Annotating provenance data for individual computations is also important for data computed once and used often like indices for reference genomes . To know how the genomic index, that is used in a mapping step of a workflow, was generated also increases the reproducibility of the entire workflow. Therefore, provenance for individual computatations is the first use case we want to tackle with our approach.
Assuming that we compute a genomic index of some reference genome with the Burrows-Wheeler Aligner (BWA): bwa index genome.fa
This results in a set of files describing the index of the genome. We can use the index in a mapping workflow. But after some time nobody can tell you which release of BWA was used to generate the index or which coworker computed the index. For individual computations one can use our tool in the following way: dataprov -i genome.fa -o genome.fa.bwt run \ bwa index genome.fa
The original command is wrapped by our software Dataprov. For single computations you have to tell Dataprov the input and output files of the computations because it can not infer it from the command. Dataprov generates a XML file genome.fa.bwt.prov whose schema is shown in Figure 1.
The overhead for the researcher consists of specifying input and output files of the operation. This is feasible for manually executing simple computations, but not for automatic workflows.
The second example of an individual computation also shows how provenance information of input files are incorporated into provenance information of output files. We map a set of reads against the reference genome we just created. The structure of the resulting XML file is shown in Listing 4. dataprov -i genome.fa.bwt -i A.fastq \ -o A.bam run \ ’bwa mem genome.fa A.fastq > A.bam’
Listing 4. Dataprov XML files inherit history of input files. <d a t a p r o v> <t a r g e t > <name>A. bam</name> [ . . . ] </ t a r g e t > <h i s t o r y > <o p e r a t i o n > [ . . .
I n h e r i t e d from f i r s t c o m p u t a t i o n , d e s c r i b i n g c r e a t i o n o f genome . f a . bwt . . . ] </ o p e r a t i o n > <o p e r a t i o n > [ . . . d e s c r i b i n g c r e a t i o n o f A. bam . . . ] </ o p e r a t i o n > </ h i s t o r y > </ d a t a p r o v>
This use case is in principle the same as the first one, but this time BWA is provided through a Docker container. We show only the first computation, because it’s analogue to the use case before. docker pull biocontainers/bwa dataprov -i genome.fa -o genome.fa.bwt run \ docker run -v $PWD/:/tmp/:z -it \ docker.io/biocontainers/bwa:latest \
bwa index /tmp/genome.fa
The computation takes now place in a Docker container. Dataprov interpretes the docker command and extracts the container name. Then it uses the python docker package to inspect the container and creates a XML element describing the operation and the container as shown in Listing 5.
We implemented support for Snakemake workflows into our application to demonstrate the main purpose of our work, the automatic generation of provenance metadata during workflow execution. This means you can run Snakemake workflows whilst generating provenance data for the resulting files. The
Listing 5. Docker commands have their own operation class. Not only the wrapped command and details about used Docker version is stored, but also valuable information about the Docker container. <opClass> <docker> <command>d o c k e r run [ . . . ] < / command> <d o c k e r C o n t a i n e r > <imageSource> <d o c k e r L o c a l>
d o c k e r . i o / b i o c o n t a i n e r s / bwa : l a t e s t </ d o c k e r L o c a l> </imageSource> <i m a g e D e t a i l s > <imageID>sha256 : 0 b01483 [ . . . ] < / imageID> <repoTag>d o c k e r . i o / b i o c o n t a i n e r s / bwa : l a t e s t </repoTag> <r e p o D i g e s t > [ . . . ] < / r e p o D i g e s t > <c r e a t e d >2018 02 08T11 : 0 4 : 5 6 . 7 5 6 2 5 0 0 8 9 Z</ c r e a t e d > <l a b e l s > <i t e m key =” s o f t w a r e ” v a l u e =”bwa”/> <i t e m key =” s o f t w a r e . v e r s i o n ” v a l u e =”0.7.15”/ > </ l a b e l s > </ i m a g e D e t a i l s > <d o c k e r P a t h >/ u s r / b i n / docker </ d o c k e r P a t h> <d o c k e r V e r s i o n>
Docker v e r s i o n 1 . 1 2 . 6 , b u i l d 88 a4867 / 1 . 1 2 . 6
</ d o c k e r V e r s i o n>
</ d o c k e r C o n t a i n e r >
</docker>
</ opClass>
actual use case is composed of the workflow from the
snakemake tutorial3. The workflow searches for genomic variants
in sequencing data. The inputs are two sets of reads which
are then mapped to the reference genome using the
BurrowsWheeler Aligner (BWA) [
To run the workflow and call all variants, one would execute the following command: snakemake all.vcf
This runs the Snakemake workflow until the specified target file is created. The workflow is described in a so-called Snakefile that specifies rules. Rules have files as targets and Snakemake builds a directed acyclic graph (DAG) from the rules until the specified target file is reached. Then Snakemake executes the commands inferred from the DAG.
For creating the provenance data one has to execute the following command: dataprov run snakemake all.vcf
Compared to the execution of individual commands the overhead here is very small. Internally Dataprov performs a dryrun of the Snakemake workflow to infer which rules are executed and extract the corresponding commands. Finally the workflow is executed and the resulting provenance files are generated. Because the resulting XML file comprises over 200 lines and can be viewed in the example directory of the GitHub repository4.
3http://snakemake.readthedocs.io/en/stable/tutorial/tutorial.html 4https://github.com/fbartusch/dataprov/tree/master/examples/snakemake
Dataprov is versatile and can be used on a local workstation, a HPC cluster, or a virtual machine in the cloud. The tool does not require any special privileges and can be integrated into tools and workflows offered by science gateways.
Science gateways offer convenient access to tools and
workflows. An example for such a science gateway is the
MoSGrid portal [
If a science gateway integrates the proposed method the gateway can return the provenance metadata together with the computational results. The tools and workflows offered by the science gateway are still convenient to use and in addition all data processing steps are traceable for the scientist.
As shown in the use cases the implementation has almost no overhead for the operator of the science gateway if certain workflow engines are used to perform the computation.
The tool noWorkflow [
The tool ReproZip [
CodaLab 5 offers a collaborative platform for reproducible research. One can upload data and Docker containers to a CodaLab server and run experiments on the data using an environment provided by Docker containers. Data and runs are stored on the platform as immutable bundles. Worksheets written in a custom markdown language present an experimental pipeline and can contain references to the immutable bundles. Because whole research process takes place on the CodaLab platform the experiments are well documented and reproducible. The difference to Dataprov is that a researcher moves the computational work to the CodaLab platform whereas Dataprov runs in the familiar to the researcher.
Several universal provenance models were created in the
past. Two of the most common are the Open Provenance
Model (OPM) [
Many workflow engines like Snakemake or Galaxy are able to generate a graphical representation of a workflow and return also the commands that were executed during workflow execution. The additional benefit of Dataprov is, that the tracked operations can consist of entirely different types. This means for example that the preprocessing of data can be done via a Snakemake workflow whereas the key results are generated by another script or even manual shell commands and the provenance information is still available in a well defined format.
The presented method is able to wrap single shell commands as well as entire Snakemake workflows and produces a human and machine readable provenance metadata. The provenance metadata describes the transformation of the input data to the output data in detail and thus enhances the reproducibility of the computation massively.
We also discussed how the integration of the proposed metadata schema could improve data provenance of computations executed through science gateways.
Wrapping single computations on the command line has the overhead of specifying input and output of the computation. The support of workflow engines reduce this overhead to the minimum. This has to be implemented for each workflow engine individually. The implementation effort depends heavily on the workflow engine and its capabilities. Engines providing an easy to use API or even a dryrun functionality are easier to support.
The resulting XML metadata file is well defined by a schema and can be validated. For computations with fewer steps the XML document is with not much effort readable by a human. The file tends to be very long and thus not so easy readably if the computation comprises many steps, input files, and output files. Thus a visual representation of the metadata would give more insight into the provenance information. This could be accomplished by generating a simple directed acyclic graph showing data files as nodes and the executed commands as edges between the nodes. A more sophisticated visualization could comprise an interactive graph that shows additional information about files, executors, and environment as mouseover effect.
The authors acknowledge support by the High Performance and Cloud Computing Group at the Zentrum fu¨ r Datenverarbeitung of the University of Tu¨ bingen, the state of Baden-Wu¨ rttemberg through bwHPC and the German Research Foundation (DFG) through grant no INST 37/935-1 FUGG. Part of the work presented here was also supported through BMBF funded project de.NBI (031 A 534A) and MWK Baden-Wu¨ rttemberg funded project CiTAR (”Zitierbare wissenschaftliche Methoden”).