Simulation of living systems often requires numerous computational experiments on the same model for diferent parameter values. This paper describes the design of a user-friendly workflow automation system LiFlow for simulation of living systems, which is capable of conducting such a large series of computational experiments on supercomputers. The system provides a convenient interface for preparing input experimental data, executing the experiments on a supercomputer, and storing experimental results in a storage system. A distinctive feature of LiFlow is its simplicity and usability-the system is intended to be used by researchers in mathematical biology and biophysics without extensive knowledge in parallel computing. The paper provides examples of the use of the LiFlow system for simulation of the human heart left ventricle.
Simulation of living systems requires significant computational resources. Such investigations are certain to be rather time-consuming and, thus, are hard to be conducted in a reasonable time without parallel computing systems and supercomputers. However, the use of parallel computing systems requires a high degree of qualification in computer science, which many researchers involved in living systems modeling do not possess or want. Moreover, the data preparation for computational experiments is routine and time-consuming. The user needs to copy data to a supercomputer, compile the source code if necessary, enqueue the jobs with the supercomputer resource manager, and keep track of their completion. Such routine tasks should be automated.
Living system simulation often demands a large number of computational experiments on the same model but with varying parameter values. Nowadays, researchers have to prepare the configuration, input data, and the desired parameter values, and then separately execute the simulation software for each experiment. With the number of required computational experiments increasing up to hundreds or even thousands, which is typical for living systems simulation, manual preparation and execution of experiments becomes very labor-intensive and often nearly impossible. In such a case, scientists often execute only a fraction of the required computational experiments, which negatively affects the research results, hence the need to automate the execution of series of computational experiments with varying parameter values.
Another problem that researchers in simulation of living system often face is non-reproducibility of computational experiments. This problem is directly related to the large number of computational experiments that scientists have to carry out in order to obtain meaningful results. Due to pressure for publishing, scientists devote little time to keeping the records of experimental details, especially in case of hundreds or thousands of experiments. In addition, many other factors can affect computational results, such as a change in the version of the compiler or a required library on the supercomputer. Automated recording of experimental details and storage of simulation results can help to ensure reproducibility of the computational experiments.
We developed LiFlow (LIving system simulation workFLOW), a workflow system that addresses this need for automation. LiFlow provides the scientists with a convenient graphical user interface (GIU) that allows to prepare and execute a series of computational experiments on a parallel computing system with a single click.
One of the important goals of creating the LiFlow system was to make the initial learning process of the workflow tool very simple. Otherwise, busy scientists will not invest their time in studying the capabilities of the new system, and it will be useless.
The LiFlow system is primarily intended for simulation of living systems; we provide some examples of using LiFlow to simulate the human heart left ventricle. However, LiFlow could also be used in other areas that require conducting a large number of computational experiments on parallel computing systems. 2
To bridge the gap between researchers and software engineers and reduce
experiment preparation time, scientific computation workflow systems [
Another possible solution to the problem is to use an environment that
provides the integration of application software packages with supercomputers. An
example of such system is DiVTB [
To solve the reproducibility problem special software tools can be used. They provide the ability to automatically capture and store for future use all the environment of a computational experiment, such as the simulation software, the input and output data, the hardware and software configuration of the computing system, etc.
There are two basic methods used by the reproducibility improvement tools.
One method is based on executing experiments in a virtual environment, such
as virtual machines or cloud [
The second method is based on capturing the snapshot not of the entire
virtual machine but of the simulation software executable and the output data.
This approach is used in the CDE system (Code, Data, and Environment
packaging) [
In order to ensure the reproducibility of computational experiments, we
aimed at integrating LiFlow with Sumatra [
Workflow in the LiFlow system corresponds the one shown in Fig. 1. During the first stage, researchers prepare the description of the so-called experiment series, which is a set of experiments with the same model and varying parameter values. The preparation includes the selection of simulation software that will implement the required model, generation of the configuration files with the required parameters, and creation of the input data files for each experiment. Next, the experiments are launched on a parallel computing system.
When the experiments are completed, the obtained results are automatically stored in the archive in a form ready for processing (visualization, statistical processing, etc.). Thus, a user is only required to create a description of the experiment series, all the rest is done automatically. In addition, the user is able to process the results of experiments from the archive manually using third-party tools. 3.2
Similarly to the CDE system, LiFlow uses the concept of a computational package that contains all the information required to execute a series of experiments. The LiFLow computational package consists of the following components: – Source code of the simulation software, which can be loaded from a code repository of a version control system, such as Git. – Generator of experiment series that describes how to generate the desired parameter values for the experiment series. – Initial data and parameters to launch the simulation software.
A distinctive feature of the LiFlow computational package is that it describes not one experiment but a whole series of experiments. Each experiment in the series uses the same simulation software but different values of the model parameters. The parameter values for every experiment are produced by the generators, which are a part of the software package that is based on the rules specified by the user. 3.3
The LiFlow system consists of the four main components (Fig. 2). The Computational Package Preparation Tool and the Experiment Execution GUI are installed on the researcher’s personal computer, while the Experiment Execution Engine and the Parallel Computing System Adapter are deployed to the parallel computing system.
A user creates a computational package with the help of the Computational Package Preparation Tool and uses the Experiment Execution GUI to transmit the package to the desired parallel computing system and run the experiment series. Experiment Execution Engine on the computational cluster receives the package, compiles the source code of the simulation software, and executes the generator of the experiment series to produce a set of input data files for simulation software with various parameter values. Next, the set of computational jobs is generated with the same simulation software but different input files. The jobs are queued on the computational cluster using the Parallel Computing System Adapter, which interacts with the resource manager of the cluster.
Once the job is completed, the results of the experiment are automatically recorded to the Experiment Archive on the storage system. After all the jobs in the experiment series are completed, the Experiment Execution Engine sends an email with the report on the experiments’ execution to the user.
The planned Sumatra module would be able to capture the environment of the computational experiment and store it in the Experiment Catalog in order Experiment Execution
GUI Computational Package
Preparation Tool Source Code Repository
LiFlow System
Experiment Execution Engine Parallel Computing
System Adapter Parallel Computing System
Experiment Catalog
Sumatra
Experiment Archive to share the initial data and simulation results among the researchers and to improve the experiments’ reproducibility. 4
The first stage of the LiFlow system implementation has been currently completed. The computational package in the implementation is represented by a directory in a file system that contains the subdirectories with the following components: the source code of the simulation software, the generator of the experiment series, the initial data for the generator, and the script for executing the experiments.
In the current implementation, the generator is a script that creates a series of experiments by varying the parameters in the configuration file of the simulation software. The LiFlow system supports two options for specifying the parameter values: – The range of the parameter: an initial value, a final value, and an increment.
One record in a configuration file of the generator produces the input data for several experiments. – The explicit parameter values declaration. The parameter values must be specified for each experiment in the series.
The prepared computational package is transferred to the parallel computing
system using the SSH or SFTP protocols. Next, the source code of the simulation
software is built on the computational cluster. If the build process fails, LiFlow
warns the user and sends back to him the build log file. In the case of a successful
compilation, the system runs the generator of the experiment series to produce
the input data for the experiments.
Currently, only one version of the Parallel Computing System Adapter is
implemented, which is based on the SLURM Workload Manager [
The scripts in the LiFlow system are written in Python. The storage of simulation software source codes is implemented as a Git repository provided by a third-party service.
Users are provided with a simple graphical interface, which allows one to execute a series of experiments on a parallel computing system in one click (Fig. 3). The user needs to select the parallel computing system to perform the computation, specify the credentials (login and password), the path to the folder with the computational package, and the email address (for job completion notifications). When the user clicks the Launch button, the LiFlow system starts the workflow process. The text output shows the current stage of the process of setting up the experiment and, if an error occurs, specifies where did it happen. Fig. 3 demonstrates an example of a successfully submitted experiment. The LiFlow GUI is also written in Python using the PyQt4 library and is designed to work both on Windows and Linux.
A disadvantage of the current LiFlow implementation is the lack of a failover mechanism. If an error occurs, the experiment will not be repeated. This approach is chosen because the failure can be caused not only by problems with hardware or system software, but also, more frequently, by an error in the simulation software or a wrong combination of parameters, for which the computation cannot be performed. In such a case, restarting the experiment will not lead to solving the problem, it will only unnecessarily load the computational cluster. Still, a failure in carrying out one experiment does not lead to termination of the entire experiment series. 5
Nowadays, the LiFLow system is integrated with the URAN supercomputer at
the Krasovskii Institute of Mathematics and Mechanics and the computational
cluster of the Ural Federal University. The system had been used on these clusters
to execute several experiments in simulation of human heart left ventricle (LV)
using the LeVen simulation system [
The study of the influence of the fiber direction in the LV anatomical model
on the speed and consistency of its electrophysiological activation was performed
using the LiFLow system [
The same system can be used to reproduce the results of the research
manually conducted before. Two series of experiments were performed in the
investigation of the excitation speed of the LV myocardial tissue by using an anatomical
model that allows to change the shape of the ventricle and the direction of the
fiber course in it [
The paper [
The users of the LiFlow system, researchers in mathematical biology and biophysics from the Institute of Immunology and Physiology UrB RAS, provided a generally positive feedback. Before, they needed approximately 30 minutes to manually prepare and run one computational experiment on a parallel computing system. With the help of the LiFLow system, they could execute a series of dozens or hundreds of experiments in less than one hour. The users appreciated the convenience of the LiFlow GUI and the ability to obtain the results of simulation from the storage system. As a result, they do not have to deal with the Linux operating system on the computational cluster, which is unfamiliar to them. Overall, LiFlow helped the researchers from the Institute of Immunology and Physiology UrB RAS to conduct computational experiments more efficiently.
As opposed to the popular scientific computation workflow systems such as Taverna, Kepler, and Triana, which provide for building large, complex computational workflows, the LiFLow system provides only one simple workflow. However, this limitation provides an opportunity to make the LiFlow system extremely easy to use. The complicated computational workflow systems are especially useful in domains with standardized data formats and tools, such as bioinformatics (the Taverna system is specifically targeted at bioinformatic applications). However, adding new components into such workflow systems is rather difficult. In contrast, the LiFlow system is more suitable for researchers who write the simulation code by themselves. Unfortunately, due to the beta version of our project, we will be able to publish the source code later. 7
The paper presents the LiFlow computational workflow system intended to automate the processing of a large number of computational experiments for living systems simulation on parallel clusters. Distinctive features of LiFlow are the automatic generation of the input data and parameters for carrying out experiment series. The system has been used for simulation of the human heart left ventricle. The use of LiFlow can significantly reduce the preparation time of a series of experiments, as well as make processing of their results more convenient.
Directions for future work include: – Full integration with the Sumatra tool to ensure the reproducibility of launched experiments. – Developing the mechanisms of secure integration of several computational clusters from different organizations with a single LiFLow instance in order to share computational resources and simulation results. – Implementing the Parallel Computing System Adapters for cluster resource managers other than SLURM, as well as for cloud. – Creating more advanced and flexible generators of experiment series integrated with GUI.
Acknowledgments. The work is supported by the RAS Presidium grant I.33P “Fundamental problems of mathematical modeling”, project no. 0401-2015-0025, and by the Research Program of Ural Branch of RAS, project no. 15-7-1-26. Our study was performed using the Uran supercomputer of the Krasovskii Institute of Mathematics and Mechanics and computational cluster of the Ural Federal University.