Adapting scientific workflows to changing infrastructures Ninon De Mecquenem1 , Supervised by Ulf Leser1 1 Humboldt-Universität zu Berlin, Unter den Linden 6, Berlin, DE 10099 Abstract Scientific workflows are increasingly popular for large-scale data analyses as they promise better documentation, increased reproducibility, and easier scalability of complex analysis pipelines. However, reproducibility is severely reduced when a given workflow is optimized for a specific infrastructure, as it would require other scientists to access the same computing environment. Hence, it is important to develop techniques that automatically adapt a given workflow to changes in the underlying infrastructure or characteristics of the analyzed data, for instance, by using different data partitions or different tools for individual steps of the analysis. Automatic workflow adaptation requires a cost model setting properties of different tools, data set sizes, and characteristics of the given infrastructure into perspective. As a first step in this direction, we here study in detail the performance of an important analysis in genomics, namely RNASeq, in different settings. We experimentally measured the runtime of different RNAseq workflows implemented in Nextflow on different infrastructures (stand-alone or distributed), composed of different tool chains, using different data set sizes. As different tools also lead to (slightly) different outputs, we additionally compared the output of different workflow variants. We show that workflow variants designed for a given infrastructure perform much worse in other settings and that rewritings sometimes keep and sometimes change the output, even when tools are only replaced by others with the same purpose. We see these experiments as an important first step toward automatically adapting workflows to different infrastructures. Keywords Data Analysis Workflows, Bioinformatics, Distributed infrastructures, Portability 1. Introduction data and the operations - the executing infrastructure can- not make any assumptions regarding the functionality Data Analysis Workflows (DAWs) are used to solve a of the operations nor the format of the data [7]. Sec- specific data analysis problem using a chain of tools con- ond, DAWs for complex scientific data analysis consist of nected by input/output dependencies. In bioinformatics, many steps that are heuristics, which means that the ”cor- the usage of DAWs is critical to perform reproducible rect” result of an analysis actually is not known and that analyses [1]. However, porting DAWs to different infras- different DAWs for the same purpose on the same data tructures or using them for different input data sizes can might produce diverging results [8]. Therefore, DAW cause severe problems. For example, if the new infras- adaptation may consider a wide range of valid primitive tructure has fewer resources, the workflow can crash operations, such as: the replacement a tool of the DAW due to insufficient memory, or time outs as computations by another one with the same purpose (1), the change of take longer than anticipated at workflow design time. On tool/DAW parameters (2), the modification of the DAW the other hand, also with more resources scaling prob- structure (3), or the adjustment of the sizes of data parti- lems can affect the runtime of the analysis [2, 3, 4]. In tions (4). Before implementing such functionalities, it is our research, we hypothesize that knowledge of the in- crucial to understand the impact of a given adaptation for frastructure, the input, the DAW itself and the particular a given setting on the workflow runtime and the output. tools it is made of can be used to automatically adapt In this paper, we study this problem for DAWs perform- a given DAW such that it performs gracefully also in a ing an RNA sequencing (RNAseq) analysis. RNASeq is new environment. The rewriting of chains of interde- particularly interesting as it is a widespread of analysis pendent commands has a long tradition, especially in used to understand gene expression and regulation under the database [5] and the big data world [6]. However, certain conditions or diseases such as cancer. RNASeq rewriting DAWs for scientific data analysis differs from DAWs take a large set of short strings as input, which are these settings in two regards. First, DAWs are typically sequenced fractions of mRNA, the transient molecules designed and executed in a black box model both for the generated during gene expression as an intermediate step to protein sequences. DAWs next map each string to a Published in the Workshop Proceedings of the EDBT/ICDT 2023 Joint reference genome to then cluster sets of strings stemming Conference (March 28-March 31, 2023, Ioannina, Greece) from the same transcript. Real-life DAWs also include Envelope-Open mecquenn@informatik.hu-berlin.de (N. D. Mecquenem); further steps, such as data pre-processing, quality filter- leser@informatik.hu-berlin.de (S. b. U. Leser) Orcid 0000-0003-3052-6129 (N. D. Mecquenem); 0000-0003-2166-9582 ing, or computation of different quality metrics. Each (S. b. U. Leser) task of these DAWs can be performed by several tools © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). that serve the same purpose, but use different heuristics CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 1 Ninon De Mecquenem et al. CEUR Workshop Proceedings 1–4 leading to different resource requirements and results. the field of AutoML [13]. The main differences are that A particularly complex step is the mapping, for which a in AutoML (1) pipelines are linear and only exchanges of plethora of possible tools exist [8]. tasks are considered and (2) the results of different work- Here, we studied the behaviour of three RNAseq DAWs flow variants may also vary, but that typically a notion of on two different infrastructures using two different data ”best” is defined (e.g. highest accuracy on a test data set), sets. We created the DAWs tool-chains based on the tools’ which often is not the case in scientific data analysis. popularity and compatibility with each other. Each DAW was implemented in two versions: one is designed for a stand-alone server, and another one is designed for a distributed infrastructure. Adaptation consists of split- ting the load of resource-demanding tasks across several nodes of the cluster by splitting the input files - which is not supported equally well by all tools. We ran the two versions of these three workflows on two different infrastructures and measured the runtime and output differences between the workflow versions depending on several parameters. We consider this work as a base to better understand the impact of DAW rewritings. Ulti- mately, we aim at abstracting these findings into a set of rules that lead to an automatic DAW rewriting according to a given input and context specifications. 2. Related work Figure 1: Design of the experiments. RS1 was created with three tool-chains (Salmon, STAR, Hisat2). A variation of these Several strategies have already been used to optimize workflows (RS2) was created. We ran it on two infrastructures DAW execution in a given context. For instance, [9] sur- (infra1 and infra2) and two datasets (D1 and D2). Their run- veys genomic workflows for the Map-Reduce infrastruc- time was measured, and their output was compared. ture. Zaid Al-Ars et al. created a version of the popular RNAseq GATK workflow adapted to Spark [10]. Yakeen et al. describe a large-scale variant caller optimized for execution on a commercial cloud [11]. Roy et al. stud- 3. Experiments ied the influence of different Hadoop parameters on a specific genomics DAW [12]. However, all these works fo- As described in Figure 1, we created two RNASeq work- cused on optimization for a specific target infrastructure; flows performing the same operations but optimized for in contrast, we research methods that can adapt DAWs different infrastructures. Both have as central and most to any execution infrastructure. Another line of related time-consuming task the alignment, which matches short research is concerned with optimization of (relational) stretches of genomic sequences to a reference genome. queries with user-defined functions, especially in big The alignment is also fundamental in other bioinformat- data processing pipelines [6]. These, however, typically ics workflows studying genomic sequences [14]. RS1 are built on the paradigm that (a) individual operations follows a pipeline structure, which we assume fits bet- (tasks in a workflow setting) have pre-defined semantics, ter to a stand-alone server. RS2 is optimized for a dis- (b) data follows a relational model, and (c) all rewritings tributed infrastructure, as it splits the input to allow for preserve exactly the results of a query. These assump- a distributed computation of the alignment step across tions do not hold in the realm of DAWs for scientific several nodes of a cluster. From each workflow, we fur- analysis - data can have arbitrary formats, tasks are typi- thermore created three variants according to the spe- cally exchanged as binaries without any guarantees, and cific tool used to compute alignments, i.e., STAR [15], operations are partly computationally so complex that Salmon [16], and Hisat2 [17]. We implemented all DAWs they can only be approached using heuristics, leading to using Nextflow [18], a workflow engine of increasing different results for different concrete physical implemen- popularity in the Bioinformatics community. Nextflow tations. Accordingly, we envision that DAW rewriting workflows are implemented in a specific DSL which al- makes up for these more complex settings by relying on lows for automatic parallelization and distributed execu- knowledge provided by workflow designers that want to tion of tasks. For local execution, NextFlow uses its own support portability and re-usability of their workflows. execution engine; for a distributed setup, it can work Finally, there is some commonality to recent studies in together with Kubernetes resource managers. The goal of this experiment is 2 Ninon De Mecquenem et al. CEUR Workshop Proceedings 1–4 Data and Infrastructures DAW Dataset Infrastructure RS1 RS2 Infra1 58 m 48 m D1 Two RNAseq-paired input datasets of different sizes were Infra2 364 m 134 m STAR considered. Both data sets were obtained by sequencing Infra1 401 m 321 m D2 the transcriptome of Drosophila melanogaster. The dif- Infra2 2383 m 720 m ference in size allows us to understand better the impact Infra1 60 m 47 m D1 of the input size on the decision to rewrite the workflow. Infra2 175 m 85 m Hisat2 Infra1 569 m 232 m Dataset 1 consists of two paired files of 13GB each, and D2 Infra2 935 m 437 m Dataset 2 is two files of 48G. Infra1 8m 15 m The DAWs were run on two different infrastructures: D1 Infra2 68 m 51 m one stand-alone server and one cluster. The stand-alone Salmon Infra1 32 m 51 m server (infra1) consists of 32 Intel Xeon CPU E5-2667 v2 D2 Infra2 186 m 270 m Octa Core, with a memory of 387 GB and a SATA SSD 1,9TiB Raid 5. The cluster Infra2 in our experiments con- Table 1 sists of 10 homogeneous nodes, each with a Quadcore Runtimes of the three RNAseq DAWs, compared across DAW versions and datasets for both infrastructures. The DAWs are Intel Xeon CPU E3-1230 V2 3.30GHz; Memory: 16 GB; named by the aligner used in their tool-chain. In bold, we Disks: 3x1TB, connected by a network of 2x 1GBit. The highlight large reduction in runtimes from RS1 to RS2. stand alone server has way more resources than the clus- ter. Therefore, we expect all the runtimes to be faster on Infra1. Interestingly, the DAWS require very different run- times depending on which tool was used for the align- 4. Results ment step. On Infra1 with D2, the runtime of Salmon is approx. 20 times faster than HiSat2 on RS1 and five times Runtime comparison faster on RS2. In almost all cases, workflows profited from the plus in resources in the distributed Infra2 when Table 1 shows the runtimes of the pipeline version (RS1) switching from RS1 to RS2, but to varying degrees. and the distributed version (RS2) of the DAWs on the two However, recall that our intention is not to find the best infrastructures with both datasets. Note that each value DAW for a given infrastructure, but to develop algorithms displayed in this table was obtained from a single run. that can rewrite a given DAW developed for a setting A We are currently generating duplicate runs to acquire to adapt it to a new setting B - which might simply have more robust values. However, as we had exclusive usage a slow network, such as Infra2. For instance, imagine a of the infrastructures during the measurements, we are researcher developed RS1 on Infra2 using HiSat2 for data confident about our experiments not being perturbed by sets of the size of DS1. Now, she wants to run it on larger other computations. Furthermore, the duplicated runs datasets yet avoid that 5-fold increase in runtime. An that were already computed are consistent with the re- adaptation an optimizer could propose is to rewrite the sults presented in the table. workflow into RS2, which would only lead to a 2-fold run- We observe notable runtime differences that show ten- time. Or imagine another user who wants to reuse this dencies but not an entirely consistent picture. In general, workflow, but is forced to use STAR as aligner because it RS1 and RS2 show similar runtimes on the stand-alone is the lab-internal standard. Runtime would be doubled, server Infra1, with the notable exception of Hisat2 on the or even increased by a factor of 13 when also switching to large dataset D2. For this case, time reduction is almost larger files. An optimizer could recognize that switching 50%, while runtimes for the smaller dataset D1 are very to RS2 would decrease the expected increase by 65%. similar. We attribute this behaviour to the low resource usage of Hisat2. In a non-distributed setting, Nextflow parallelizes the tasks over the different CPUs available, Quality comparison which makes the runtime on Infra1 overall smaller. Al- In scientific data analysis, different DAWs for the same most no difference is observed for STAR on infra1 as it problem often lead to (slightly) different results due to requires a lot of RAM to run over a single chunk of the the usage of different heuristics for solving complex sub- input data. On Infra2, runtimes differ considerably. In al- problems. Sometimes, avoiding such changes can be most all cases, RS2 (designed for distributed computation) mandatory, for instance, when a certain analysis method achieves much lower runtimes than RS1, with reductions is defined as an organizational standard. However, often up to 66%. Again, there is one exception: Salmon actu- such changes are acceptable, for instance, in the early ally takes longer with RS2 than with RS1. This runtime phases of a data analysis project in which different trade- difference is due to the task splitting the input files. offs are explored, such as runtime, result quality, analysis cost etc. In any case, users need to be informed about the 3 Ninon De Mecquenem et al. CEUR Workshop Proceedings 1–4 STAR Salmon Hisat2 Scientific Workflows in NGS Data Analysis: A Case Dataset 1 100 % 82,06 % 97,39 % Study, arXiv:2006.03104 (2020). Dataset 2 100 % 71,71 % 99,65 % [3] F. Lehmann, D. Frantz, S. Becker, et al., FORCE on Table 2 Nextflow: Scalable Analysis of Earth Observation Percentage of similarity of transcript counts between the data on Commodity Clusters, CIKM Workshops pipeline version (RS1) and the scatter/gather version (RS2) (2021). of each DAW. [4] M. Hanussek, F. Bartusch, J. Krüger, Performance and scaling behavior of bioinformatics applications in virtualization environments to create awareness expected degree of changes a DAW rewriting would incur. for the efficient use of compute resources, PLOS To this end, we compared the results of the different DAW Computational Biology 17(7): e1009244 (2021). versions to understand how much the DAG structure [5] H. Garcia-Molina, J. D. Ullman, J. Widom, Database modification impacts analysis results. We measured the systems - the complete book, Pearson (2009). similarity between the results of the RS1 and RS2 versions [6] A. Rheinländer, A. Heise, F. Hueske, et al., Sofa: of each DAW in Table 2. Clearly, the DAWs using Hisat2 An extensible logical optimizer for udf-heavy data and STAR are very robust to this rewriting, while the one flows, Information Systems 52 (2015) 96–125. using Salmon produces largely different results. [7] R. Mork, P. Martin, et al., Contemporary challenges for data-intensive scientific workflow management systems, Workshop on Workflows in Support of 5. Conclusion and future work Large-Scale Science (2015). We presented the results of an initial study on the impact [8] A. Schaarschmidt, A. Fischer, E. Zuther, et al., Eval- of DAW adaptations to different infrastructures, consid- uation of seven different rna-seq alignment tools ering both the replacement of central tools as well as based on experimental data from the model plant changing the workflow structure. The main purpose is arabidopsis thaliana, Int J Mol Sci (2020). to show that such rewritings impact performance con- [9] Z. Quan, L. Xu-Bin, J. Wen-Rui, et al., Survey siderably and that certain variants are more suitable for of MapReduce frame operation in bioinformatics, certain infrastructures and that suitability also depends Briefings in Bioinformatics 15 (2013) 637–647. on the input size. Relationships are overall complex and [10] A.-A. Zaid, W. Saiyi, M. Hamid, Sparkra: Enabling certainly will vary with different analysis problems, dif- big data scalability for the gatk rna-seq pipeline ferent DAWs for solving them, and different infrastruc- with apache spark, Genes 11 (2020). tures. We are consolidating these results with experi- [11] S. Yakneen, S. Waszak, M. Gertz, et al., Butler en- ment replicates and more workflows and dataset sizes. ables rapid cloud-based analysis of thousands of In future work, we will focus on languages to provide human genomes, Nat Biotechnol (2020). descriptions of core aspects of infrastructures, methods [12] A. Roy, Y. Diao, U. Evani, et al., Massively parallel to derive properties of tools on different infrastructures, processing of whole genome sequence data: An annotation schemes to describe the equivalence of tools in-depth performance study, SIGMOD (2017). in genomics and a cost model as a basis for a rule-based [13] K. He, Xin amd Zhao, X. Chu, Automl: A survey of DAW adaptation algorithm that takes these properties the state-of-the-art, Knowledge-Based Systems 12 into account. We will then develop an automatic DAW (2021). rewriting that implements this algorithm. [14] R. Musich, L. Cadle-Davidson, M. Osier, Com- parison of short-read sequence aligners indicates strengths and weaknesses for biologists to consider, Acknowledgements Front Plant Sci. (2021). [15] A. Dobin, C. A. Davis, F. Schlesinger, et al., STAR: Funded by the Deutsche Forschungsgemeinschaft – ultrafast universal RNA-seq aligner, Bioinformatics Project-ID 414984028 – SFB 1404 FONDA 29 (2012) 15–21. [16] R. Patro, G. Duggal, M. Love, Salmon provides fast and bias-aware quantification of transcript expres- References sion, Nat Methods (2017). [1] L. Wratten, A. Wilm, J. Göke, Reproducible, scalable, [17] D. Kim, J. Paggi, C. Park, Graph-based genome and shareable analysis pipelines with bioinformat- alignment and genotyping with hisat2 and hisat- ics workflow managers, Nat Methods (2021). genotype, Nat Biotechnol (2019). [2] C. Schiefer, M. Bux, J. Brandt, et al., Portability of [18] P. D. Tommaso, M. Chatzou, E. Floden, Nextflow enables reproducible computational workflows, Nat Biotechnol (2017). 4