Workflow systems support scientists in capturing computational experiments and managing their execution. However, such systems are not designed to help scientists create and track the many related workflows that they build as variants, trying different software implementations and distinct ways to process data and deciding what to do next by looking at previous workflow results. An initial workflow will be changed to create many new variants thereof that differ from each other in one or more steps. Our goal is to support scientists in the iterative design of computational experiments by assisting them in the creation and management of workflow variants. In this paper, we present several use cases for creating workflow variants in hydrology, from which we specify requirements for workflow variants. We also discuss major research directions to address these requirements.
workflow variants, computational
Scientific workflow systems play a major role in supporting scientists to design, document and execute their computational experiments, automatically tracking provenance during the workflow execution [11; 1]. Scientists follow an iterative exploratory cycle where they often create an initial workflow, and then explore variations of it using different data, replacing some of the software steps, or adding new steps. Sometimes workflows have to be modified because of changes in data (e.g. when datasets are updated with new formats) or software (e.g., software is no longer available, a newer version is better).
In current workflow systems, scientists manage this exploratory process manually. Updating a workflow is a complex and timeconsuming task that may involve several steps, and may require tracking down information about different versions of the software used in the workflow. 2
Computational workflows describe the computational steps and the dataflow among them to perform complex multi-step analyses. The steps are implemented by software components (or workflow components) that process data. A software component has a welldefined interface consisting of input and output files as well as parameter constant values. The dataflow between components is captured as connections among their respective interfaces. A workflow component may be implemented by a scientist, for example a routine to check for erroneous sensor readings. A workflow component may also be implemented using third-party software, for example invoking a linear regression function from a machine learning software package. A workflow component can be updated in two ways. In some cases, a new upgrade of the component is created to override a previous one, for example in cases where the underlying software was corrected to fix a bug. In other cases, a new variant of the component is created with new inputs or outputs or other modifications, where the previous versions are still valid and available to the user to use in workflows.
Workflow executions are the result of running workflows and provide provenance for the newly generated data products.
After running a certain workflow, a scientist may want to explore a workflow variant that represents a variation of an existing workflow that was run earlier where one or more steps are changed. That step change may require changing other steps that may be affected. In other cases, the scientist may create a new workflow upgrade of a previously run workflow that simply replaces a component by a new one with a bug fix. When a workflow is upgraded, the scientist may need to redo previous runs.
Due to the exploratory nature of science, a scientist may start with an initial workflow and iteratively create many workflow variants. During this process, the scientist will want to consider different designs of variants, compare any given variant with previous ones, and synthesize the results of several variants with comparable settings. This iterative process of creating and managing workflow variants is currently not well supported. There are several reasons why a scientist may create a workflow variant: 1. New versions of the software used in the workflow components are released. These may add new functionality that could be useful for the investigation. These may also correct errors or fix bugs, and in that case the scientist may be interested to check whether their results done with the older version still hold. 2. New possible models or algorithms become available.
The scientist may discovery these through online search, reading articles, or talking with colleagues. These open new possibilities for exploring alternative designs of the workflow. 3. New datasets become available to the scientist. In this case, the scientist may want to change their workflow to incorporate that new kind of data.
Sometimes the explorations are due to a combination of these. For example, new software versions may fix errors and offer new functionality that allows the scientist to use new kinds of data.
There have been several efforts to keep track and manage workflow
updates and versions. VisTrails [
Koop et al. [
Workflow variants are also explored in Experiment Lines [
This section describes several scenarios where scientists iteratively
create and explore workflow variants. The scenarios use examples
from hydrology. A hydrologist uses models, often developed by
others, to estimate how much water will flow in an area. We will
consider several hydrology models in these scenarios.
MODFLOW is the U.S. Geological Survey's three-dimensional
(3D) finite-difference groundwater model that has been developed
for several years and has many versions and variants. A major
version of the core implementation is MODFLOW-2005 [
Hydrology models need data about the area for the simulation. For example, MODFLOW requires elevation data, in the U.S. typically coming from the National Elevation Dataset, recharge data, typically from the National Recharge Dataset, and the data for the area from the Watershed Boundary Dataset.
Figure 1 shows an initial workflow W0 that uses MODFLOWNWT. The input data includes the boundary for the area being studied, elevation data, and recharge (drainage) data. Area is an input to the step Rasterize, which converts the data from a geographic system format to raster (bitmap) and is implemented using GDAL (Geospatial Data Abstraction Library). The step Convert Area converts the raster data to text data (ASCII) using NumPy, a library for array/matrix in Python. Then, the unit of measurement for this data is converted from centimeters to meters and the data format is converted from raster to text using the step Convert Elevation, which is implemented using NumPy. Rasterize Recharge, which is implemented using GDAL, generates the recharge raster and then Convert Recharge, which is implemented using NumPy, converts the unit of measurement of the data from centimeters to meters and from meter/year to meter/day and from raster format to ASCII. The simulation component is implemented using MODFLOW-NWT Version 1.0.2, and uses FloPy to configure it with the appropriate packages. 4.1
In this case, a workflow component is replaced by another one that uses a different version of software to implement it but the component interface remains the same. We consider two main scenarios for this case. One occurs when a new version of the software used in a workflow component is released to fix errors or bugs. The other one occurs when a new version is released to carry out a different function.
The first scenario S1 starts with a scientist that runs workflow
W0 several times, changing the data sets used and comparing the
results to understand how changes in the inputs influence the
results. After several weeks, the scientist notices a new release of
MODFLOW, with modifications to enhance the model outputs. So
they create an upgrade of the MODFLOW component, which
results in the creation of workflow W1, an upgrade of W0, shown in
Figure 2. In this example, from version 1.0.2 to 1.0.3 of
MODFLOW-NWT a bug was fixed in the UZF1 package that was
causing UZF1 to incorrectly calculate unsaturated-zone
evapotranspiration, which results in a much smaller value [
The second scenario S2 occurs when the software in a component is modified to carry out a different function. In this case, from version 1.0.2 to 1.0.3 of MODFLOW-NWT there is a major change to generate a more accurate calculation of evapotranspiration. First, the header of the listing file that results from running MODFLOW-NWT is changed from having a variable "RMS" to "RMS1" and "RMS2," and from a variable "L2-NORM" to "L2-NEW" and "L2-OLD". This change was done to improve the calculation of the residual terms as the L2-NORM rather than the root-mean-squared error (RMS error). This change does not affect the format of the results, only their value to be more accurate. Thus, the interface of the new component variant does not change, and the newly created workflow variant W2 has the same structure as W1 in Figure 2.
The scientist needs to be able to understand the changes to the software in new versions in order to assess the differences between versions and estimate the effort to make the changes in the workflow. This may require a significant effort, as this information may be scattered across release notes, documentation, papers, web sites, and other sources. In some cases, the scientist may be interested in skipping ahead several versions. For example, she may want to change from version 1.0.2 all the way to the newest version that is 1.1.3. This is challenging since the scientist needs to track and summarize all the differences between several consecutive versions.
In addition, when changing a software version used to implement a component, the scientist needs to check if the new version is compatible with the software version of other components of the same workflow. For example, a specific FloPy version is compatible only with some MODFLOW-NWT versions. Sometimes these incompatibilities can occur across different workflow components, for example if two components make different assumptions about the Newton-Raphson formulation. This means that the scientist needs to track in detail all the software dependencies and compatibilities across the software components of a workflow.
Finally, the scientist may need to check that the new simulation results do not require additional changes in the workflow steps that use those results. In our case they were the output of the workflow, but in other cases further adjustments may be required.
Scenarios 1 and 2 motivate the following requirements: R1 – Version descriptions need to capture useful metadata of the software. R2 – Scientists need to understand differences in metadata between different software versions, particularly about their interfaces. R3 – Scientists need to be alerted about relevant updates of software used in their workflows. R4 – Workflow descriptions need to capture the software, software version, and functions used in the implementation of workflow components. R5 – Scientists need to understand how new workflow variants can be used to correct errors in prior results. R6 – Scientists should be able to easily replace a component of a workflow with a new one when the interfaces of the components are the same. R7 – Given a software package that can be used to create many workflow components, scientists need to easily figure out how to implement new variants of a workflow component with newer versions of that package. R8 – Scientists should be able to easily create new versions of workflow components and relate them to each other. R9 – Scientists should be able to easily create new workflow variants and relate them to each other. R10 – Scientists should be able to relate changes in software to specific workflow results, so it is clear how new software versions affect calculated variables to produce wrong values. R11 – Version descriptions need to capture bug fixes and known bugs and relate them to software features and input and output file variables. In this case, a workflow variant is created by replacing a workflow component by another one that uses the same software implementation but invokes a different function and as a result has a new component interface (i.e., adding, removing or replacing ,whi inputs or outputs). This interface change may require changes in other steps of the workflow (e.g., adding, replacing or removing data conversion or post-processing steps.) We consider two scenarios for this case. One occurs when a component is changed to use additional inputs or outputs provided by the software used to implement it. Another occurs when a component is changed to replace inputs or outputs or use them differently in the software used to implement it. In both cases, the rest of the workflow may be affected by the changes.
Scenario S3 starts with a scientist running workflow W2. The scientist would like to add an input regarding water elevation through wells, so she creates a new variant of the workflow component by adapting the MODFLOW component used in W2 by adding a new input for well data. The well data is already provided as an ASCII file, so unlike the elevation and recharge data there is no need to convert wells data to ASCII. The only data preparation needed is converting the unit of measurement from feet to meters. To perform this change, the scientist adds Well to the workflow inputs, adds the step Convert Well for unit conversion. This results in workflow variant W3, shown in Figure 3. The scientist created one new component variant and created a variant of an existing component.
In scenario S4, the scientist decides to include snowmelt in the simulation. This can be done by using infiltration as an input instead of recharge (since the infiltration package will also account for recharge). Figure 4 shows the resulting workflow variant W4 where the recharge input of W3 is replaced with infiltration. Additional changes include replacing the steps to prepare data for simulation with those steps to clip and resample infiltration, and to convert the unit of measurement in the input data from centimeter per year to meters per day, reformatting it to ASCII format. In addition, the MODFLOW step needs to be modified in two ways. First, the recharge input needs to be replaced with infiltration input. Second, the FloPy software configures MODFLOW to use the infiltration packages. In total, the scientist created five new components and created a variant of an existing component.
There are several important tasks that the scientist needs to address in these two scenarios.
Before creating the data preparation components for W3 and W4 the scientist has to find whether components that already do those conversions are available or not. Reusing components saves time, but after spending many years running similar workflows with similar data it may be hard to remember which components have been created before. Furthermore, in addition to reusing components it may be possible to reuse entire sub-workflows. In our example, the sub-workflow to prepare infiltration data has five steps that can be reused together.
Another important task is to compare the results of different workflow variants. For example, a scientist would run W3 and W4 and compare the results to each other and to W2 to understand how changes in the workflows affect the simulation results.
Scenarios 3 and 4 motivate these additional requirements: R15 – Scientists need to easily find software packages and workflow components that are appropriate to process a specific type of data input. R16 – Scientists need to easily find workflow components for data conversion. R17 – Scientists need to be able to understand the differences between two workflow variants. 4.3
In this case, a workflow variant is created by replacing a workflow component by a component that does an equivalent function but is implemented using a different software. There are several reasons to create workflow variants that use equivalent software, such as testing different models or taking into account parameters that are ignored by the current model used in a workflow. The new component may have a very different interface from the previous one, thus requiring a major update of the workflow to create, replace, or remove several data preparation or post-processing steps. Note that although the tasks to create the workflow variant may be similar to those in scenarios 3 and 4, now there are additional tasks in finding out information about the new software to check its functionality and analyze how it fits into the workflow and the overall exploration that the scientist is doing. We consider two scenarios. One occurs when the scientist already knows which alternative method to use. Another one occurs when the scientist needs to find and compare the assumptions, functionalities and the effort to change the workflow when considering more than one method to decide which one to use.
Scenario S5 has a scientist who is concerned about MODFLOW-NWT only solving for saturated zones, so some parameters are simplified or even ignored. The scientist would like to use a different method to solve for the unsaturated zones, and has heard that the MIKE-SHE model does a similar simulation to MODFLOW, but is a fully coupled and integrated surface water and ground water model that considers parameters regarding unsaturated zones. The inputs and outputs for MIKE-SHE are different from MODFLOW. MIKE-SHE uses area data in the same raw format that is provided by the data source, so it does not need to be pre-processed. It also uses topography data and can ingest formats very similar to the data source, so the data only needs to be clipped. MIKE-SHE also requires several new input data, namely rainfall, evaporation, and temperature, all in the format provided by data sources so they only need to be clipped. MIKE-SHE also generates several separate outputs, including files associated with the simulation (SHERES), a binary output file containing all the static information on the simulation (FRF), and other results stored in a series of DFS0, DFS2 and DFS3 files. As for the MIKE-SHE component, there is no need to use the FloPy software to implement it. Figure 5 shows workflow variant W5 created from W4: the scientist had to create three new data pre-processing components, one new component for MIKE-SHE, and one new component to combine the simulation results to obtain a format that is comparable to W4 and to all the previous workflow variants. Many components from W4 were discarded as they were no longer needed.
In scenario S6, the scientist decides to investigate other models,
since MIKE-SHE is a commercial, proprietary model. There are
many other hydrology models available, including PIHM [
It is important to highlight several important tasks done by the
scientist in these scenarios. In both scenarios, but particularly in
scenario S6, the scientist needs to compare how two models are
similar and how they differ in terms of the input data that they use
and the output data that they generate. The documentation of
models always includes details of the input and output requirements
in terms of files and formats. The scientist will want to understand
conceptually how the models work in terms of the physical
variables used or generated in the model. That is, understanding
the inputs and outputs at the file and format level is important, but
understanding how model variables map to each of the files is also
necessary. This information is usually not included in the software
documentation, but in the publications associated with the model.
The scientist will need to consult a variety of sources in order to
understand how different models compare [
Another important task is to understand the assumptions made by the different models. For example, in hydrology some models may assume the Navier-Stokes equations for fluid motion, while others do not. These assumptions are often not captured in the descriptions of workflow components, which focus on the models as software artifacts rather than research artifacts.
In addition, after creating and running the new workflow variants W5 and W6, the scientist will want to compare their results to the results obtained with previous workflows W4, W3, and earlier ones. This requires that the scientist understands how the model results are related to one another, which requires understanding what modeling variables are generated and included in the simulation outputs.
Scenarios S5 and S6 motivate the following additional requirements: R18 – Version descriptions need to capture assumptions used in software. R19 – Workflow components, inputs, outputs or parameters in new workflow variants that are no longer needed need to be removed. different versions of the software and the different versions and variations of a given workflow component. Workflow updates, which address the creation of new workflow variants by replacing, adding, or removing workflow components, the propagation of the effects of those changes throughout the structure of the workflow, and the validation of the new workflow variants. Workflow comparisons, which address the comparison between different software versions, software packages, workflow variants and workflow runs.
Table 1 summarizes the requirements introduced in this section,
pointing out the broad categories they belong to and the cases where
they occur. Although we adopt the hydrology domain in our
scenarios to illustrate the requirements, our requirements are
domain-independent. Workflows in any domain have
preprocessing steps, post-processing steps, and major analytic steps
[
Given the state of the art and the requirements from the scenarios, we outline possible research directions for future work: Describing workflow
components and their underlying
software. This includes the creation and adaptation of existing
ontologies to capture information about software versions and
variants, including software interfaces and features. OntoSoft
[
and tracking
workflow This includes how to variants compare and
their
workflow
components and workflow variants regarding their interfaces
and functions, and present these results in a useful way for
scientists to understand their differences and the implications
on experiment results. A possible approach is using
multimedia
narratives
that
combine
text,
graphics,
and
visualizations to explain the similarities and differences
between software versions, software variants, workflow
versions, or functions/methods. More importantly, these
narratives should be easily customized to the reader’s level of
expertise and interest. As a starting point our approach may be
based in an approach for data narrative generation [
Designing an interactive framework to support scientists in 5 This paper discusses the need to support scientists in exploring different experiment designs over time.
We presented several scenarios where an initial workflow is modified to create workflow variants by replacing, adding or removing workflow steps. describe the requirements of these scenarios, and grouped them into categories: updates, and workflow comparisons. We also discussed major research directions to address those requirements, including improved frameworks for describing workflow components and the associated software, for managing and tracking workflow variants, and supporting scientists in the iterative exploration and experimentation process through workflow variants.
Acknowledgments. This work was supported in part by a grant from the US National Science Foundation under award ICER-1440323 and ICER1632211 (EarthCube RCN IS-GEO), and in part by the Sao Paulo Research Foundation (FAPESP) under grants 2017/03570-3, 2014/23861-4 and 2013/08293-7.