Scienti c discoveries are increasingly dependent upon the analysis of large volumes of data from observations and simulations of complex phenomena. Scientists compose the complex analyses as work ows and execute them on largescale HPC systems. The work ow structures are in contrast with monolithic single simulations that have often been the primary use case on HPC systems. Simultaneously, new storage paradigms such as Burst Bu ers are also becoming available on HPC platforms. In order to maximize the performance of data analyses work ows today it is critical to determine the characteristics of the work ows. Obtaining a deeper understanding of the work ows helps us identify opportunities to leverage the capabilities of the Burst Bu er. In this paper, we analyze the performance characteristics of the Burst Bu er and two representative scienti c workows. We measure the performance of these work ows using the Burst Bu er, allowing us to make recommendations for future optimal usage of work ows using Burst Bu er.
The science drivers for high-performance computing
(HPC) are broadening with the proliferation of high-resolution
observational instruments and emergence of completely new
data-intensive scienti c domains. Scienti c work ows that
chain the processing and data are becoming critical to
manage these on HPC systems. Thus, while providers of
supercomputing resources must continue to support the
extreme bandwidth requirements of traditional
supercomputing applications, centers must now also deploy resources
that are capable of supporting the requirements of these
emerging data-intensive work ows. In sharp contrast to
the highly coherent, sequential, large-transaction reads and
writes that are characteristic of traditional HPC
checkpointrestart workloads [
I/O needs of data-intensive work ows to ensure that correct resources can be deployed with the correct balance of performance characteristics.
The emergence of data-intensive work ows has coincided
with the emergence of ash devices being integrated into
the HPC I/O subsystem as a \Burst Bu er", a
performanceoptimized storage tier that resides between compute nodes
and the high-capacity parallel le system (PFS). The Burst
Bu er was originally conceived for massive bandwidth
requirements of checkpoint-restart workloads for extreme-scale
simulation [
Today's commercially available Burst Bu er solutions [
In this paper, we characterize two of the production data analytics work ows used at the National Energy Research Scienti c Computing Center (NERSC) at Lawrence Berkeley National Laboratory, and we present an analysis of their performance on the production Burst Bu er resource deployed as a part of NERSC's Cori system. The paper is organized as follows. Section 2 presents the background for the paper - related work and the details of the NERSC Burst Bu er Architecture. Section 3 details our approach to scalable I/O characterization for both work ows and Section 4 presents a detailed analysis of the I/O requirements of these work ows. We discuss e cient use of Burst Bu ers in Section 5 and provide conclusions in Section 6. 2.
In this section we describe related work and the NERSC Burst Bu er architecture. 2.1
Scienti c Work ows. Data-intensive scienti c work ows
have been shown to process large amounts of data with
varied I/O characteristics [
Burst Bu ers. Several uses of Burst Bu ers have been
shown in order to mitigate the I/O bottlenecks of
dataintensive workloads [
NERSC's Cori system features a Burst Bu er based on
Cray DataWarp [
Cray's DataWarp middleware aggregates the SSDs on each of the BB nodes and provides user jobs with dynamically provisioned private parallel le systems. Users can request a certain capacity of Burst Bu er in 200 GiB increments (which we call fragments) when submitting jobs. Each fragment is allocated on a di erent BB node to allow the aggregate performance of the BB allocation to scale with the requested capacity. DataWarp also designates one of the BB nodes as the metadata server for the allocation. This allocation is mounted on the job nodes when the job is launched, and it is typically torn down upon job completion. However, users may also request a persistent mode allocation, which allows a BB allocation to persist across multiple jobs.
DataWarp also o ers private mode reservations where each compute node gets its own metadata server within the Burst Bu er allocation and, by extension, its own private namespace. This enables higher aggregate metadata performance since each compute node's metadata is serviced by a unique BB node. 3.
In this section, we detail our performance analysis methodology and workloads used for our analyses. 3.1
The two work ows studied in the paper were selected because they stress the I/O subsystem in very di erent ways: CAMP is limited by metadata performance and SWarp is limited by data transfer performance. When discussing the work ows, we use the term \work ow pipeline" to refer to a single unit of the larger work ow. 3.1.1
The CAMP (Community Access MODIS Pipeline)
workow processes Earth's land and atmospheric data obtained
from MODIS satellite data [
The CAMP work ow consists of two processing steps { a) builddb, that assembles and maps swaths to their corresponding sinusoidal tiles and b) reproject, that converts the MODIS products from a swath coordinate system to a sinusoidal tiling system. Figure 1 shows the high-level representation of the CAMP work ow that includes the data staging operations to and from the Burst Bu er. The workow pipeline in this paper transforms one MODIS product's swath coordinates for one day into one speci c tile. CAMP is written in Python and generates an intermediate SQLite database to provide the mapping for the reproject stage. We use Conda, which uses the Anaconda Python distribution, to install CAMP on DataWarp. 3.1.2
The SWarp work ow combines overlapping raw images of the night sky into high quality reference images. It is used in the Dark Energy Camera Legacy Survey (DECaLS) to produce high quality images of 14,000 deg2 of northern hemisphere sky. In this survey, each SWarp work ow pipeline produces an image for a 0.25 deg2 \brick" of sky. The average input to each work ow pipeline is 16 32 MiB input images and 16 16 MiB input weight maps.
The SWarp work ow pipeline consists of a data resampling stage and a data combination stage. The data resampling stage interpolates the raw images and creates resampled images which can be trivially stacked. The data combination stage reads back the resampled images and then performs a reduction over the pixels to produce a single stacked image. The raw, resampled and stacked images are all in Flexible Image Transport System (FITS) le format. The DAG when using a Burst Bu er is similar to CAMP: input images and weight map les are staged-in prior to the data resampling stage and the combined image is staged-out after the data combination stage. SWarp is written in C and multithreaded with POSIX threads. 3.2
The work ow pipelines are run in their production con
guration on Cori and all I/O is directed to DataWarp mount
points. The DataWarp reservation is con gured to use a
shared namespace and one fragment of capacity. A job
reservation is used for SWarp and a persistent reservation is used
for CAMP (in order to retain the CAMP Python software
environment between jobs). The Integrated Performance
Monitoring (IPM) pro ling tool [
The high-level characteristics of the stages in a single work ow pipeline are shown in Table 1. The work ow stages are found to spend 10 - 30 % of time in I/O. This is the best achievable I/O time and can only get worse as more workow pipelines contend for the same storage resource.
Figures 2 and 3 show how I/O time changes with concurrency for the most time-consuming stage of each work ow. I/O time is divided into time spent in metadata operations and data operations. The experiments are repeated three times at each node count and the plots show the mean time per work ow pipeline stage. The error bars simply show the range of mean times over the three experiments.
Figure 2 shows the scaling of SWarp-resample. The results show that wall clock time remains relatively constant until about 16 work ow pipelines and that I/O time is domSWarp rsmpl
SWarp coadd
CAMP db
CAMP
reprj
Compute threads
I/O threads
Wall time (s)
I/O time (s)
I/O time (%)
Peak mem. (MiB)
Total le size (MiB)
16
1
inated by data rather than metadata operations. Figure 3
shows the scaling of CAMP-builddb is limited by metadata
performance. One source of these metadata operations is
from the startup of Python applications, which is known to
be a scalability issue in Python HPC applications [
In this section, a) we discuss the key characteristics of the work ows analyzed and use the information to highlight the e ective use of Burst Bu ers and, b) we apply this knowledge to explain how to achieve the optimum performance with the DataWarp implementation of a Burst Bu er. 5.1
The key ndings from our experimental analyses are: 1. A single work ow pipeline does not provide the I/O parallelism needed to make e cient use of Burst Bu ers. The data analytics work ows studied in this paper consist of single-process applications which perform I/O with a single thread of execution. This is poorly matched with the need to have multiple I/O streams to obtain the peak performance from Burst Bu er Flash storage. Unfortunately, single I/O stream work ow pipelines are a common feature of high throughput data analytics work ows. We show that better utilization of Burst Bu er resources is possible by executing multiple concurrent work ow pipelines against the same unit of Burst Bu er storage. Our results indicate that a single unit of DataWarp storage on Cori can sustain the I/O requests from approximately 16 concurrent work ow pipelines before there is any slow down. 2. A scaled out work ow pipeline is often limited by metadata performance. Our analysis has found signi cant metadata costs originating from database transactions, Python initialization and opening many small les. The aggregated metadata operations from multiple work ow pipelines can easily saturate a single metadata server, as shown in the CAMP-builddb work ow stage. 3. It is valuable to explicitly control the data in the Burst Bu er tier. The work ows read input data sets and produce a number of intermediate les which can be discarded once there are nal results, e.g. the resampled images in SWarp and the SQLite database in CAMP. Therefore, we do not expect automatic le movement between the Burst Bu er and the PFS to bene t these two data analytics work ows. This is because the one-time cost of staging the input data at access time may not be hidden by signi cant data reuse. Automatic le movement would also transfer the intermediate les to the PFS unnecessarily. 4. It is valuable to leave data in the Burst Bu er tier for longer than a single batch job. We have found that input les and software environments are reused across work ow pipelines.
The input data for data analytics work ows are generally Write Once Read Many times (WORM). In the SWarp work ow a single input image often contributes to multiple regions of the sky. Therefore it is wasteful to re-stage the same input le multiple times for each work ow pipeline.
gle work ow pipeline. In the CAMP work ow the Python environment is responsible for some of the I/O. The role of \support I/O" (e.g. Python packages) is rarely mentioned in the context of Burst Bu ers. It is useful to stage the software environment once to avoid the overhead and wear of repeatedly staging the software environment. Long-term data residency is not a good t for today's Burst Bu ers because they do not provide data redundancy. This imposes a data management burden upon the developer. 5.2
DataWarp storage reservations on Cori consist of multiple storage fragments of size 200 GiB. The scaling studies show that both SWarp and CAMP are limited by DataWarp performance rather than capacity. SWarp and CAMP have an aggregate capacity requirement of up to 2.6 GiB and 150 MiB per work ow pipeline, respectively (Table 1). However, the performance saturates before fully utilizing the 200 GiB of capacity at approximately 16 work ow pipelines per DataWarp fragment. This means that excess capacity must be reserved to sustain performance in a scaled out work ow. Metadata bottlenecks, such as seen in CAMP-builddb, can be addressed by combining the reservation of excess capacity with the private mode feature of DataWarp. 6.
In this paper we analyzed the performance of two scienti c work ows running on the Cori supercomputer with the DataWarp Burst Bu er. We show that a single workow pipeline does not have the parallelism to utilize the capabilities of the Flash storage hardware. We also show that the work ows have di erent I/O performance characteristics: SWarp is bound by data transfer performance and CAMP (speci cally CAMP-builddb) is bound by metadata performance as the work ows are scaled out. The results are used to give general advice about using Burst Bu ers more e ciently and to provide speci c advice for DataWarp.
This work was supported by Laboratory Directed Research and Development (LDRD) funding from Berkeley Lab, provided by the Director, O ce of Science and O ce of Science, O ce of Advanced Scienti c Computing Research (ASCR) of the U.S. Department of Energy under Contract No. DEAC02-05CH11231. This research used resources of the National Energy Research Scienti c Computing Center, a DOE O ce of Science User Facility supported by the O ce of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. The authors would also like to thank Rollin Thomas for help with installing the CAMP Python software environment on DataWarp.