Many scientific workflows are data-intensive and need be iteratively executed for large input sets of data elements. Reducing input data is a powerful way to reduce overall execution time in such workflows. When this is accomplished online (i.e., without requiring users to stop execution to reduce the data and resume execution), it can save much time and user interactions can integrate within workflow execution. Then, a major problem is to determine which subset of the input data should be removed. Other related problems include guaranteeing that the workflow system will maintain execution and data consistent after reduction, and keeping track of how users interacted with execution. In this paper, we adopt the approach “human-in-theloop” for scientific workflows by enabling users to steer the workflow execution and reduce input elements from datasets at runtime. We propose an adaptive monitoring approach that combines workflow provenance monitoring and computational steering to support users in analyzing the evolution of key parameters and determining which subset of the data should be removed. We also extend a provenance data model to keep track of user interactions when users reduce data at runtime. In our experimental validation, we develop a test case from the oil and gas industry, using a 936-cores cluster. The results on our parameter sweep test case show that the user interactions for online data reduction yield a 37% reduction of execution time. • Massively parallel and high-performance simulations.
Scientific Workflow Management Systems (SWMS) with parallel
capabilities have been designed for executing data-intensive
scientific workflows, scientific workflows for short hereafter, in
High Performance Computing (HPC) environments. A typical
execution may involve thousands of parallel tasks with large input
datasets. When the workflow is iterative, it is repeatedly executed
for each element of an input dataset. The more the data to be
processed, the longer the workflow may take, which may be days
depending on the problem and HPC environment [
Data reduction can be accomplished in at least three different
forms. First, users can do it before the execution starts. However,
in most complex scenarios, the high number of possibilities makes
it impossible to know beforehand the uninteresting subsets,
without any prior execution. Furthermore, not only the initial
dataset can be reduced, but also the data generated by the
workflow, since the activities composing scientific workflows
continuously produce significant amounts of partial data that are
consumed by other activities. A second form of data reduction is
to do it online. When the SWMS allows for partial result data
analysis, the user may interact with the generated partial data, find
which slice of the dataset is not interesting, and reduce the dataset
online. We use the term online for the interactions where users are
able to inspect workflow execution, analyze partial and
performance data, and dynamically adapt (i.e., steer) workflow
settings while the workflow is running (i.e., at runtime). The third
form of data reduction is by stopping execution, reducing the data
offline, and then resuming execution with the reduced dataset.
Because of the difficulty in defining the exploratory input dataset
and the long execution time of such iterative workflows, users
frequently adopt the third form. However, in the offline form, the
SWMS is not aware of the changes, and the results with one
workflow configuration are not related to the others. Therefore,
this is generally more time-consuming, there is no control or
registration of user interactions, and the execution may become
inconsistent [
Online data reduction has obvious advantages but introduces
several problems related to computational steering in HPC
environments [
Our approach is to represent workflow input datasets as database
relations. Each input element from the scientific domain dataset is
represented as a tuple of the input relations. When the elements of
the input dataset are files, we insert paths to these files. The
approach is implemented in Chiron, a parallel SWMS that adopts
a tuple-oriented algebraic approach [
Paper organization: Section 2 gives a motivating example. Section 3 gives the background for this work. We present our online data reduction approach in Section 4 and our adaptive monitoring approach in in Section 5. Section 6 gives the experimental validation. Section 7 discusses related work. Section 8 concludes.
In ultra-deep water oil production systems, a major application is
to perform risers’ analyses. Risers are fluid conduits between
subsea equipment and the offshore oil floating production unit.
They are susceptible to a wide variation of environmental
conditions (e.g., sea currents, wind speed, ocean waves,
temperature), which may damage their structure. The fatigue
analysis workflow adopts a cumulative damage approach as part
of the riser's risk assessment procedure considering a wide
combination of possible conditions. The result is the estimate of
riser’s fatigue life, which is the length of time that it will safely
operate. The Design Fatigue Factor (DFF) may range from 3 to
10, meaning that the riser’s fatigue life must be at least DFF times
higher than the service life [
Sensors located at the offshore platform collect external
conditions and floating unit data, which are stored in multiple raw
files. Offshore engineers use specialized programs (mostly
complex simulation solvers) to consume the files, evaluate the
impact on the risers in the near future (e.g., risk of a fracture
occurrence), and estimate the risers’ fatigue life. Figure 1 presents
a scientific workflow composed of seven piped specialized
programs (represented by workflow activities) with a dataflow in
between.
Critical Case Selection (Activity 4) and Curvature
Critical Case Selection (Activity 5) calculate the fatigue
life of the riser based on the stresses and curvature, respectively.
These two activities filter out results corresponding to risers that
certainly in a good state (no critical stress or curvature values
were identified), which are of no interest to the analysis.
Calculate Fatigue Life (Activity 6) uses previously
calculated values to execute a standard methodology [
Mattoso et al. [
Online provenance data management in SWMS is an essential asset to support all six aspects of computational steering in scientific workflows. In this section, we explain the three computational steering aspects explored in this paper.
We address two aspects of workflow data that should be
interactively analyzed: (A) domain dataflow and (B) workflow
execution information [
A. Domain dataflow. Workflows are composed of activities
(scientific programs, scripts, or services) linked as a dataflow.
Each activity invocation, or task, may consume input datasets and
input raw data files and produce output datasets and files. These
flows form the domain dataflow. To support domain dataflow
interactive analysis, Chiron stores dataflow provenance data in the
wf-Database during execution and makes them available for
online user queries. Users can then query the wf-Database using a
query interface or SQL following PROV-Wf [
Chiron’s tuple-oriented engine first stores input dataset as tuples
in the wf-Database. In parameter sweep, tuples are data values
from the Cartesian product of input parameters. Then, each task
consumes input tuples retrieved from this database, executes
them, and then stores the generated output tuples in the
wfDatabase immediately after task execution, adequately
maintaining the data relationships to the input tuples. The
workflow activities that generated the tuples are also stored in the
wf-Database and linked accordingly. Large raw data files
consumed or produced by each task are not stored in the
wfDatabase, but are rather linked to them, for file flow management.
Thus, Chiron enables online fine-grained domain dataflow
analysis [
B. Workflow execution information. Lower level execution engine
information, such as physical location (i.e., virtual machine or
cluster node) where a task is being executed, can highly benefit
data analysis and debugging in HPC execution. Users may want to
interactively investigate how many parallel tasks each node is
running. Moreover, tasks can run domain applications that result
in errors. If there were thousands of tasks in a large execution,
how to determine which tasks resulted in domain application
errors and what the errors were? This also eases debugging, an
important feature to be provided in large parallel executions.
Furthermore, performance data analysis is very useful. Users are
frequently interested in knowing how long tasks are taking. All
this workflow execution information is important to be analyzed
and can deliver much more interesting insights when linked to
domain dataflow data. When execution data is stored separately
from domain and provenance data, these steering queries are not
possible or demand combining different tools and writing specific
analysis programs [
To support all this, Chiron’s wf-Database registers parallel workflow execution data. This means that all necessary execution information for the parallel engine to work are linked to domain dataflow data and managed in the same database. Table 2 shows some provenance queries for the Risers Analysis workflow that link workflow execution data to domain dataflow.
Another important form to help gaining insights from the data is
by monitoring in a passive way. It means that users can set up
some monitoring analyses and wait for the monitoring results to
be generated. Results might be delivered to end-users as graphical
dashboards or three-dimensional in-situ scientific data
visualizations. As users gain insights from monitoring results, if
the SWMS has dynamic analytical support, they can adapt
previously set up monitoring configurations or add new
monitoring analysis [
If the SWMS allows for provenance data analysis during workflow execution, monitoring becomes more effective, since the SWMS can exploit the continuously populated wf-Database. By doing this, all the aforementioned data provenance analysis and queries executed by users may be used by a monitoring engine.
After users have analyzed partial data and gained insight from
them, they may decide to adapt the workflow execution.
Adaptation brings powerful abilities to end-users, putting the
human in full control of scientific workflow executions. Many
aspects can be adapted by humans, but very few systems support
human-in-the-loop actions [
Populating the wf-Database during workflow execution can help
all these aspects. In the Chiron SWMS, it has been shown that it
particularly facilitates steering. For example, in [
Since provenance data is so beneficial, we consider that when a user interacts with the workflow execution, new data (user interaction data) are generated, and thus their provenance must be registered. In a long-running execution, many interactions can occur and many adaptations may be made. If the SWMS does not adequately register the provenance of interaction data, the users can easily lose track of what they have changed in the past. This is critical if the entire computational experiment takes days to finish and many specific adaptations had to occur, since it may be impossible for the users to remember in the last day of execution what they have steered in the first days. Furthermore, adding interaction data to the wf-Database enriches its content and enables future user interaction analysis. One example of how such data can be exploited is that the registered interaction data could be used by artificial intelligence algorithms for understanding interaction patterns and recommend future adaptations. For these reasons, the SWMS that enables computational steering should collect provenance of user interaction data. To the best of our knowledge, this has not been done before.
In this section, we show our main contribution. Suppose that after analyzing the monitoring results, a user identifies, within the entire dataset, the subset that contains those values can be removed, hence reducing the dataset.
However, reducing a dataset to be processed has specific
constraints that need to be addressed so the execution remains in a
consistent state, i.e., with valid data and with guarantee that the
execution will not crash. As previously described, Chiron
implements a relational data model in a tuple-oriented algebraic
approach for scientific workflows [
We propose to represent the input dataset as a database relation,
which is a set of tuples. In the tuple-oriented approach [
Addressing which specific subset will be removed is quite
important. So, we first formalize the subset to be removed
(Section 4.1). In Section 4.2, we describe how we implemented
this in d-Chiron, which is a modified version of Chiron that
manages the wf-Database in an in-memory distributed database
system and is significantly more scalable than the original Chiron
[
In the tuple-oriented approach, to address a subset of the dataset to be removed, we first define a slice, which is a subset of tuples to be removed according to a criteria defined by the user. Let with data schema ₰ = {} be the relation that represents a workflow activity input dataset to be reduced. {} is the set of attributes !, 1 ≤ ≤ | |, from and each ! assumes a data type (integer, string, boolean, etc). Moreover, we split into two subsets and , where is the subset of containing tuples that have already been processed and is the subset of containing tuples that will be processed. Thus, ← ∪ | ∩ = ∅. and have the same schema ₰.
Then, we define a slice § as a subset of , which is represented as
a primary horizontal fragment of , defined by the selection
relational algebraic operation [
Once the user has selected the slice to be removed (based on userdefined criteria), the slice can be cut off. For this, we define the operation using the difference relational algebraic operator, so that , § ← − §.
By doing so, we ensure that only tuples from will be removed, since § only contains tuples from . This is necessary because only tuples that have not been processed yet (i.e., they are “ready” to be processed) can be removed. Otherwise, either the data or the workflow execution may become inconsistent. We note that our solution is applicable to reduce input data of any workflow activity that needs to process a dataset, as long as the SWMS is aware of the data elements composing the dataset.
In this section, we describe how we implement slice removal and
cut in d-Chiron. In the SWMS that implements the tuple-oriented
approach and manages execution data in the wf-Database, each
input tuple (or set of tuples, depending on the dataflow operator)
to be consumed is related to a task. For this reason, removing
tuples means removing the tasks that would consume the
referenced tuples. In the PROV-Wf data model [
To ease slice removal in d-Chiron, we developed a Steer
module. With the Steer module, users can issue command lines
to inform the name of the input relation and the formula to
select the slice to be removed. Then, the module is responsible for
retrieving the identifiers !"#$% of the ready tasks to be removed
(analogous to the set needed for the slice definition). Instead of
physically removing the tasks from the wf-Database, we choose to
mark them with the state REMOVED_BY_USER. By doing so, we
enable these tasks to be later analyzed with provenance queries
and to be consistently related within the table Modified_Task.
To guarantee consistency, we take advantage of d-Chiron’s
database system [
The wf-Database tables are distributed, thus making concurrency
control of the Task table partitions even more complex. In
dChiron, distributed concurrency control in the Task table is
outsourced to the distributed database system that guarantees the
ACID properties [
In this section, we combine monitoring and computational steering into an adaptive monitoring approach. Our workflow monitoring approach relies on online queries to the continuously populated wf-Database. Users can set up monitoring queries (as in Table 1 and Table 2) and analyze monitoring results.
In Section 5.1, we present a formal description and describe the implementation in Section 5.2 with the extensions to PROV-Wf to accommodate adaptive monitoring and online data reduction.
Monitoring works as follows. There is a set {} composed of monitoring queries !, 0 ≤ ≤ | |, each one to be executed at each ! > 0. Users do not need to specify queries at the beginning of execution, since they do not know everything they want to monitor. This is why {} starts empty. After users gain insights from the data, after some interactive provenance data analyses, they can add monitoring queries to {} in an ad-hoc manner. Each ! can be adapted by users, meaning that users have control of the time frame of each ! during execution. Each ! execution generates a monitoring query result set !", = !| ∈ ℕ!! , at each time interval !. We constrain that each !" must deliver one column only. If users want more columns, they can write different monitoring queries for each new column. However, the number of rows in the result set is not limited. This means that each monitoring result set !" should be either a scalar value or an array.
To improve human-in-the-loop, the end-users have the flexibility to adapt monitoring during workflow execution. To do so, at each instant after each monitoring query result !" has been generated, the values for ! and ! are reloaded from the wfDatabase. If any change has happened, it will be considered in the next iteration + !. Moreover, at each certain amount of time during execution (also configured by the user), the system checks if the user has added new monitoring queries in {}. Our adaptive monitoring approach takes full advantage of the data stored online in the wf-Database. More importantly, it enables users to dynamically steer monitoring settings (including which data will be monitored and how), highly benefiting them in finding uninteresting subsets to be removed.
To implement our approach, we first need to extend the wfDatabase schema. To store {}, we add the table Monitoring_Query, shown in Table 4.
Similar to what we did for the Steer module, we also developed the module Monitor to facilitate utilization. The Monitor should start at any cluster node that is able to access the distributed database system and should start after the workflow execution has begun, whenever users want to monitor the workflow execution. Similar to what we did for the Steer module, we also developed the module Monitor to facilitate utilization. The Monitor should start at any cluster node that is able to access the distributed database system and should start after the workflow execution has begun, whenever users want to monitor the workflow execution. A command line starts the Monitor module that runs in background. It establishes a connection with the distributed database system (connection settings are provided in the XML configuration file). Chiron (and d-Chiron) makes use of this XML file to define the workflow design, workflow general settings, and other user-defined variables. Then, the Monitor program keeps querying the Monitoring_Query table at each to check if a new monitoring query was added. The default value for is 30s, as the time interval to check if monitoring queries were added or removed. However, users can customize this. After the Monitor has started, users can add (or remove) monitoring queries to (or from) the Monitoring_Query table. Currently, users can add monitoring queries using a command line to inform the SQL query to be executed at each time interval and the time interval. Whenever the Monitor module identifies that the user added a new monitoring query, it launches a new thread. Each thread is responsible for executing each monitoring query in Monitoring_Query at each defined time interval. A thread is finished when a monitoring query is removed or when the workflow stops executing (in that case, all threads are finished). Figure 3 shows the steps executed at each time interval.
2. Store query results in the wf-Database 3. Reload all information for ! from the wf-Database for the next time iteration. The user could have adapted any of this information.
To enable all these monitoring capabilities and human-adaptation, three of these steps represent queries to the wf-Database, including reads and writes. The stored results can be further analyzed a posteriori or, more interestingly, used as input for runtime data visualization tools, since results are immediately made available after they are generated.
Another contribution of this paper is that we add three concepts to
PROV-Wf [
These concepts are: UserQuery, MonitoringQuery, and MonitoringResult, as in Figure 4. Using PROV nomenclature, UserQuery is a PROV Activity that stores the user queries that remove sets of tuples and thus influence the state of the associated tasks (i.e., remove them). MonitoringQuery is a PROV Activity that contains the monitoring queries submitted by the user in specific time intervals. The monitoring queries generate PROV Entity MonitoringResult that stores the query results.
In this section, we validate our solution (for online data reduction and adaptive monitoring) based on a real data. In Section 6.1, we show the experimental setup, Section 6.2 shows a test case where an expert monitors the execution and removes slices of the dataset. In Section 6.3, we analyze the added overhead.
Scientific workflow. As a proof of concept for this work, we use
a synthetic parameter sweep workflow of the Riser Fatigue
Analysis example (see Figure 1), which is based on a real case
study. The workflow manipulates approximately 300 GB of raw
data. In all executions, we use the same dataset, which spans over
approximately 12,000 data elements to be processed in parallel.
Depending on the workflow activity, tasks may take few seconds
(e.g., Activity 7) or up to one minute on average (e.g., Activity 3).
Software. In all executions, we use d-Chiron [
Hardware. The experiments were conducted in Grid5000 using a cluster with 39 nodes, containing 24 cores each (936 cores). Every node has two AMD Opteron 1.7 GHz 12-core processors, 48GB RAM, and 250GB of local disk. All nodes are connected via Gigabit Ethernet and access a shared storage of 10TB.
Let us consider the following scenario. Peter is an offshore engineer, expert in riser analysis and learned how to set up monitoring, analyze d-Chiron’s wf-Database, and use the Steer module developed in this work. In Peter’s project, the Design Fatigue Factor is set to 3 and service life is set to 20 years, meaning that fatigue life must be at least 60 years (see from Section 2). Peter is only interested in analyzing risers with low fatigue life values, because they are critical and might need repair or replacement. During workflow execution, it would be interesting if Peter could inform the SWMS, which input values would lead to low risk of fatigue, so they could be removed. However, this is not simple because it is hard to determine the specific range of values (i.e., the slice to be removed). For this, Peter first needs to understand the pattern of input values associated to low risk of fatigue life values. In the workflow (Figure 1), the final value of fatigue life is calculated in Activity 6, but input values are obtained as output of Activity 1, gathered from raw input files. Keeping provenance is essential to associate data from Activity 1 with data from Activity 6.
To understand which input values are leading to high fatigue life values, Peter monitors the generated data online. For simplicity, we consider wind speed, which is only one out of the many environmental condition parameter values captured by Activity 1 to serve as input for Activity 2. Peter knows that wind speed has a strong correlation with fatigue life in risers. He expects that with low speed winds, there is a lower risk of accident.
When workflow execution starts, the Monitor module is initialized. Then, Peter adds two monitoring queries: ! shows the average of the 10 greatest values of fatigue life calculated in the last 30s of workflow execution, setting ! = 30s; and ! shows the average wind speed associated to the 10 greatest values of fatigue life calculated in the last 30s, also setting the query interval ! = 30s. We recall from Table 1 that ! is similar to 1, but only considering data processed in the last 30s. ! and ! queries are added to the Monitoring_Query table. Peter monitors the results using the Monitoring_Result table. These results can be a data source for a visualization that plots dashboards dynamically, refreshed according to the query intervals. After gaining insights from the results and understanding patterns, he can start removing the undesired values for wind speed. The monitoring query results !! and !! for the two previously listed queries, as well as when the user reduced the data, are plotted along the workflow elapsed time, as shown in Figure 5. It presents !! in full black line with square markers and !! in full gray line with triangle markers. These markers determine when the monitoring occurred. The workflow execution starts at = 0, but only after approximately 150s, the first output results from Activity 6 starts to be generated. From the first results, at = 150 and = 180, Peter checks that when wind speed is less than 16 Km/h (see horizontal dashed line in = 16 in Figure 5), the results lead to the largest fatigue life values. Since risers with large fatigue life values are not interesting in this analysis, he decides, at = 190, to remove all input data elements that contain wind speed less than 16 Km/h. For this, the first user query ! is issued with a command line to the Steer module. User queries are represented with circles in the horizontal axis (Elapsed time) in Figure 5. The exact time a user issued an interaction query is stored in User_Query table.
The next markers after ! happens at = 210. Comparing with the previous monitoring mark, at = 180, we can observe that this Peter’s steering (!) increases the minimum wind speed values to be considered from 14.2 Km/h to 24.1 Km/h. Also, we observe a significant decrease in the slope of the largest values for fatigue life (10.6% lower). This means that the removal of these input data containing wind speed less than 16 Km/h made the SWMS not process data containing low wind speed values, which would lead to larger fatigue life results.
Then, monitoring continues, but that slope decrease calls Peter’s attention. To obtain a finer detail of what is happening, he decides to adjust the monitoring interval time (! and !) at runtime, by reducing to 10s to get monitoring feedbacks more frequently. We can observe that for both lines !! and !!, the markers become more frequent during = [220, 270]. This is because a monitoring is registered at every 10s. We highlight that, although in test use case we are only showing monitoring correlating wind speed and fatigue life, other monitoring correlations could also be analyzed and users can add, remove or adjust monitoring queries at any time during execution.
After verifying that the results are reasonable, Peter decides to increase back the monitoring query intervals for both queries to 30s after = 270. He then observes that since !, wind speed less than 25 Km/h are leading to large fatigue life values. Then, at = 310, he calls Steer again to issue ! that removes input data for wind speed < 25 Km/h. The next markers after ! shows that this steering made the wind speed value associated to large fatigue life be at least 30.5 Km/h and a decrease of 6.5% in large fatigue life values between = 300 = 330. Similarly, Peter continues to monitor and steer the execution. He issues ! at = 370 to remove input data with wind speed < 30.5 Km/h, making a decrease of 4.9% in large fatigue life (comparing fatigue life in = 360 and = 390). Then, he issues ! at = 430 to remove input data with wind speed < 34.5, attaining a decrease of 1.7% in large fatigue life (comparing fatigue life in = 420 and = 450). Despite this small decrease, he decides at t = 520 to further remove data, but with wind speed < 35.5 Km/h. However, no decrease greater than 1% 35.0 33.0 )31.0 h /m29.0 (K27.0 ed25.0 ep23.0 s d21.0 n i 19.0 W 17.0 15.0 13.0 623 373 355 115 3 We store each interaction in the User_Query table and map (in table Modified_Task) its rows with rows in the Task table, to consistently keep provenance of which tasks were modified (in this case, removed) by each specific user steering. Thus, keeping provenance of user steering helps analyzing how specific interactions impacted the results. Figure 5 shows that some specific interactions imply significant changes in lines’ slopes. Queries on the wf-Database can show finer details about how many tuples each user interaction made the SWMS not process, as shown in Table 6. Each issued time follows Figure 5 and is registered with the timestamp of when the first activity started.
Inter act. Finally, we run the exact same workflow and input datasets, but with no monitoring or interactions to compare how such slice removals help decrease overall execution time. The workflow with no interaction processes all input data, including those containing wind speed values that lead to risers with low risk of fatigue, which are not considered in Peter’s analyses. In total, Peter’s steering yields the removal of 1469 input data elements (out of approximately 12,000). This reduces the execution time for this test case by 37% compared with no steering. Furthermore, these removed input data would make the workflow process and generate more raw data files if the input data elements were not removed. By querying the wf-Database in the end of execution, we found that the execution with no user steering processed approximately 300GB of raw data files, whereas with steering the total was 258GB, representing 14% of data reduction.
A monitoring query ! in {} is run by a thread at each ! seconds. Depending on the number of threads (|{}|) and on the interval ! there may be too many concurrent accesses to the wfq1 q2 q3 q4
Elapsed time (seconds)
q5 Steering 150 180 Database, which may add overhead. The goal of this experiment is to analyze such overhead.
We set up the Monitor module to run queries, which are variations of the queries 1-7 presented in Table 1 and Table 2. For example, in 2, we vary the curvature value. We also modify them to calculate only the results over the last seconds, at each seconds. To evaluate the overheads, we measure execution time without monitoring and then with monitoring, but varying the number of queries |{}| and the interval , which is considered the same for all queries in {} in this experiment. The experiments were repeated until the standard deviation of workflow elapsed times was less than 1%. The results are the average of these times within the 1% margin. Figure 6 shows the results, where the gray portion represents the workflow execution time when no monitoring is used; and the black portion represents the difference between the workflow execution time with and without monitoring (i.e., the monitoring overhead).
From these results, we observe that when the interval is equal to 30s, the overhead is negligible. When the interval is 1s, the overhead is higher when the number of monitoring threads is greater. This happens because three queries are executed in each time interval (see Figure 3), for each thread. In the scenarios with 30 threads, there will be 120 queries in a single time interval . In that case, if is small (e.g., = 1), there are 120 queries being executed per second, just for the monitoring. The database that is queried by the monitors is also frequently queried by the SWMS engine, thus adding higher overhead. However, even in this specific scenario that shows higher overhead (|{}| = 30 and = 1), it is only 33s or 3.19% higher than when no monitoring is used. Most of the real monitoring cases do not need such frequent (every second) updates. If 30s is frequent enough for the user, there might be no added overhead, like in this test case. We also evaluated the same scenarios without storing monitoring results in the wf-Database, but rather appending in CSV files, which is simpler. The results are nearly the same as in Figure 6. This suggests storing all monitoring results in the wf-Database at runtime, which enables users to submit powerful queries as they are generated, with all other provenance data. This would not be possible with a solution that appends data to CSV.
Considering our contributions, we discuss the SWMS with parallel capabilities with respect to human adaptation (especially data reduction), online provenance support, and monitoring features.
Although online human adaptation is the core of computational
steering, there are few parallel SWMS [
While human adaptation is less explored in parallel SWMS,
monitoring is widely supported in several existing SWMS
[
This work contributes to putting the human in the loop of scientific workflow systems, especially when users can actively steer and reduce data online to improve performance. As a solution to the input data reduction problem, we made use of a tuple-oriented algebraic approach that organizes workflow data to be processed as sets of tuples stored in a wf-Database, managed by an in-memory distributed database system at runtime. We developed a mechanism coupled to d-Chiron, a distributed version of Chiron SWMS, which allows for reducing data, while maintaining both data integrity and execution consistency. A major challenge to the problem of data reduction is to address which subset of the data should be removed. As a solution to this, we proposed an adaptive monitoring approach that aids users in analyzing partial result data at runtime. Based on the evaluation of input data elements and its corresponding results, the user may find which subset of the input data is not interesting for a particular execution, hence can be removed. The adaptive monitoring allows users not only to follow the evolution of the workflow, but also to dynamically adjust monitoring aspects during execution. We extended our previous workflow provenance data model to be able to represent provenance of the online data reduction actions by users and the monitoring results. Although we implemented our solution in d-Chiron, other SWMS could be used if provenance, execution, and domain dataflow data are managed in a database at runtime.
To validate our solution, we executed a data-intensive parameter sweep workflow based on a real case study from the oil and gas industry, running on a 936-cores cluster. A test case demonstrated how the user can monitor the execution, dynamically adapt monitoring settings, and especially remove uninteresting data to be processed, all during execution. Results for this test case show that the user interactions reduced the execution time by 37% comparing with the execution that processed the whole dataset. Although the test case was from the oil and gas domain, any other workflow application could have been used, as long as a domain expert can tell which slice is not interesting, removed with no harm to the final results.
To the best of our knowledge, this is the first work that explores user-steered online data reduction in scientific workflows steered by ad-hoc queries and adaptive monitoring, while maintaining provenance of user interactions. The results motivate us to extend our solution and explore different aspects that can be adapted by humans based on sophisticated workflow data analysis support. Our solution is currently dependent on the domain expert’s knowledge to identify correlations between input and output data to determine which subsets are uninteresting. We plan to address in-situ data visualization based on the adaptive monitoring and interactive queries results and develop recommendation models to suggest correlations based on history stored in the wf-Database. Other future work include: enabling users to set priorities to different slices of the data in a way that the SWMS system will process critic slices before; improving usability of the system by developing intuitive user interfaces to decrease the learning curve, especially related to the query interface, to take full advantage of the wf-Database. We also plan to expand our experiments and analyze how reducing each specific type of data (relation tuples, raw data files not processed and not generated) impact final results.
This work was partially funded by CNPq, FAPERJ and Inria (MUSIC project), EU H2020 Programme and MCTI/RNP-Brazil (HPC4E grant no. 689772), and performed (for P. Valduriez) in the context of the Computational Biology Institute (www.ibcmontpellier.fr). The experiments were carried out using the Grid'5000 testbed (https://www.grid5000.fr). 10. REFERENCES