I/O performance arises as a major bottleneck in nowadays dataintensive scientific applications. In order to identify execution parameters that provide an improved I/O performance, experimental efforts relying on synthetic I/O workload generators are widely employed. Focusing on addressing limitations of current workload generators, and to provide a more flexible, unified, and user-friendly approach for parallel I/O performance analysis, we have proposed a differentiated tool, called IORE. This paper focuses on demonstrating IORE applicability for I/O performance analysis of a workload derived from a real-world on scientific application. Our results indicate the potential of IORE as a parallel I/O experimental tool.
and the impact of changing some controlled parameters. For that purpose, a variety of
approaches and tools have been proposed, with synthetic I/O workload generators, such the
INTERLEAVED-OR-RANDOM (IOR) benchmark [
Focusing on addressing some limitations of the IOR benchmark and facilitating
the I/O performance evaluation task on such complex environments, we have proposed
IOR-EXTENDED (IORE) [
The remainder of this paper is organized as follows. Section 2 provides a brief overview of some related synthetic I/O workload generators. In Section 3, IORE is presented, focusing on its dataset-based workload generation capabilities. Details on scenarios and methods employed in the experiment conducted in this research work are provided in Section 4, while experimental results are discussed in Section 5. Section 6 concludes this paper with additional remarks about the observed results and planned directions for IORE development.
Synthetic I/O workload generators have been widely employed in research works focusing
on parallel I/O and storage performance for data-intensive computational science
applications [
While MPI-TILE-IO is focused on a specific dataset type, the IOR
benchmark [
Although being a de facto standard in the parallel I/O research community, some
specific dataset-based workloads are not easily or even precisely reproduced by IOR.
Further, only homogeneous workloads, in the sense of the amount of data transferred
per process, are supported in IOR. Focusing on addressing these issues, and providing a
more flexible, unified, and user-friendly approach for research works on parallel I/O and
storage performance evaluation, we have proposed the IORE I/O performance evaluation
tool [
As the name suggests, IORE was initially designed as an enhanced version of the IOR benchmark. However, the proposal of IORE goes beyond generating homogeneous offset-based I/O workloads and collecting performance metrics; it was designed to provide an experiment-oriented approach for I/O performance evaluation studies, contributing to reproducible results. Users define an IORE experiment through an intuitive JSON structure, specifying, in addition to expected workload parameters, the number of replications, order of run executions, among other options.
Also, IORE was designed focusing on extensibility, providing well specified interfaces for its main modules (Figure 1), such as Abstract File Storage Backend (AFSB) and Abstract File I/O (AFIO) interfaces. Another new feature introduced in IORE is the capability of exporting performance metrics after experiment completion. This aims at facilitating I/O performance research works, taking the burden from users of error-prone output parsing, and, thereby, reducing the time to results analysis. Finally, workload generation capabilities were also considerably extended in IORE, with the support for heterogeneous and dataset-based workloads.
Generating dataset-based workloads is of particular interest for evaluation of I/O performance on large-scale computational science simulation and scientific data visualization, as most of these applications process datasets in well-known formats. Figure 2 illustrates an example of a Cartesian dataset workload representation using IORE. The problem domain (i.e., the total dataset) in this example is composed of a three-dimensional structured grid with size 4 ⇥ 3 ⇥ 2 (dim_sizes). This dataset is distributed across 12 processes, where each process receives a part of the grid with size 2 ⇥ 1 ⇥ 1, by dividing each dimension of the dataset according to the workload definition (dim_divs). Each element of the Cartesian grid holds, in this example, three variables: an integer, a single-precision, and a double-precision floating-point (var_types). In a real-world application, these variables could refer to physical properties, such as temperature, pressure, and velocity. It can be observed in Figure 2 that the dataset in this example is stored in a single shared file, using a row-major order approach. IORE also supports accessing data in a file per process fashion, even though this parameter is not presented in Figure 2. z = 2
The dataset-based workload definition proposed in IORE, although simple, is very flexible and scalable, allowing for the definition of varying dataset sizes and shapes. Further, complex and sophisticated experiments can be conducted combining datasetbased workload definitions with other orthogonal parameters, such as different I/O APIs, middlewares, and high-level I/O libraries. This becomes a notably helpful feature for characterizing the impact of different parameters in dataset access performance, and for identifying parameters alternatives with improved performance. Moreover, the intrinsic experimental-oriented execution of IORE favors more reproducible results, leading to more accurate conclusions and more efficient optimization decisions.
To demonstrate IORE capabilities for I/O performance evaluation of dataset-based
workloads, an experiment was conducted. This experiment consists of multiple
distributed processes writing and reading a three-dimensional Cartesian dataset with size
11520 ⇥ 5760 ⇥ 94, where each element of the dataset corresponds to a single-precision
floating-point variable, to and from one or more files in a PFS. Consequently, a total
of 23 GiB of data is transferred between compute nodes and the storage system at each
write/read test. It is worth mentioning this workload is derived from a real-world
largescale scientific data visualization problem [
In this experiment, the impact of three parameters in the I/O performance was evaluated, namely, the number of processes accessing the dataset, the file mode adopted for storing the dataset, and the AFIO implementation. Table 1 lists these parameters and their respective values. As the dataset is fixed for this experiment, changing the number of processes implies in changing the dataset partitioning, which, in summary, results in less data per process. Focusing on demonstrating some optimizations provided by MPI-IO, collective I/O (col) and file view (fv) options from the MPI-IO AFIO implementation were also explored in this experiment. These two options only make sense for distributed data accesses at a shared file, and, thereby, were only evaluated for the Nx1 file mode. A full factorial experimental design was employed, resulting in 14 experiment runs, each consisting of a different combination of parameter values. The experiment was replicated two times, with the run execution order randomized across replications, favoring statistically independent measurements.
The experimental environment is composed of two clusters from the Grid’5000 testbed, in France. Data and metadata servers of the ORANGEFS PFS were deployed into the four nodes of the Hercule cluster, where each node contains two octa-core Intel Xeon E5-2620 CPUs, 32 GB RAM, and 6 TB SATA HDD. IORE was compiled with the MPICH MPI implementation and deployed at 20 nodes of the Nova cluster, each node containing two octa-core Intel Xeon E5-2620 CPUs, 64 GB RAM, and the ORANGEFS PFS served by the Hercule cluster mounted locally. A switched 10 Gigabit Ethernet network interconnects both storage and compute nodes.
Results of the experiment conducted in this research work are presented in Figure 3. The y axis refers to the average of the two experiment replications for the time taken by all processes participating in a given write/read test to conclude transferring the dataset to/from the PFS, while the x axis presents different AFIO implementations and options. Plots are organized in a grid, with upper plots presenting results of write tests, and lower plots presenting results of read tests. From left to right, plots refer to variations in the file mode and number of processes parameters, in this order: Nx1 with 20 processes, Nx1 with 40 processes, NxN with 20 processes, and NxN with 40 processes. As the variance of the response variable (i.e., latency) was very small for all experiment runs, with a coefficient of variation smaller than 10% in 13 out of 14 runs, error bars were not included in these plots for readability.
Nx1, 20 procs
Nx1, 40 procs 5509 5356
NxN, 20 procs 8150
NxN, 40 procs 7840 1634 1795 1590 1370 1311 1326 1150 7500 5000 )2500 1812 s ( y c 0 n e taL7500
Some interesting I/O performance behaviors can be observed through these results. First, the most notably result is the performance improvement observed with MPIIO when file view and collective I/O options are used together (i.e., MPI-IO (col+fv)) for either writing or reading the dataset to a shared file. Collective MPI-IO was in average 130 times faster than independent MPI-IO in write tests, and 135 times faster in read tests. These enhanced performance can be attributed to internal optimizations of the ROMIO MPI-IO implementation used by MPICH, which transforms and coordinates distributed data accesses in order to explore better performing I/O access patterns. In this particular experimental scenario, each process issues very small sized requests (i.e., 376 bytes long), a well-known undesired access pattern because of its resulting I/O performance degradation. By exchanging request information, ROMIO can perform larger data transfers, better exploring the storage system throughput capabilities.
A very contrasting behavior is observed with C standard stream I/O functions. While write performance was the worst among all results, being 3 to 7 times slower than POSIX and independent MPI-IO, C stream read performance was considerably better, being 65 to 85 times faster than POSIX and independent MPI-IO. The improved performance of read tests with C stream is a result of its buffered I/O and prefetching algorithms. At each read request, a larger part of the dataset is retrieved from the storage system, which causes subsequent request to be serviced from data cached locally. It is worth mentioning that, even though read tests happens after write tests in IORE, the read_reorder_offset parameter was set to 1 at all tests, which mean that a process i reads the data transferred by the i 1 process at each run, focusing on avoiding the impact of write cached data in read test results. Regarding write performance, these same features can have imposed an extra cost on I/O, degrading the overall performance.
Less expressive results were observed with POSIX and independent MPI-IO AFIO implementation. Independent MPI-IO was in average between 4% to 18% faster than POSIX at all experiment runs, which can be attributed to enhanced operations provided by ROMIO support to ORANGEFS. Nonetheless, without any other particular improvement, such as collective operations and buffered I/O, both APIs demonstrate a degraded performance due to the considerably large number of small I/O requests.
This paper contribution can be understood as a demonstration of some features provided by IORE, our proposal for a differentiated I/O performance evaluation tool, and how IORE can be used to assist in the analysis and improvement of the I/O performance of data-intensive scientific applications. Using IORE newly introduced dataset-based workload generation feature, an experimental study was conducted to analyze the I/O performance of writing and reading a large Cartesian dataset, whose parameters were inspired in a real-world application. Fourteen scenarios were evaluated, combining different I/O APIs and options, number of tasks, and file sharing modes.
In terms of usability, through a single and plain text experiment definition file, available at https://doi.org/10.6084/m9.figshare.6376979, IORE coordinated, executed, and exported results for all experiment runs and replications carried out in this research work. This feature alleviates the burden on users that, previously, had to prepare complex and error prone scripts and parsers to achieve the same result. Moreover, such approach favors cooperation, as experiment definitions can be easily shared and executed among different groups.
Results observed in this research work shed light on some I/O performance issues faced by real-world data-intensive scientific applications. Even for conventional and structured datasets, the performance of read and write operations can vary from dozens of seconds to hours, depending on the I/O API and options explored. Particularly, we demonstrated through this IORE experiment how MPI-IO can be highly efficient on dataset transferring using collective I/O and file view. Furthermore, results indicate that C standard stream I/O is a likely alternative for read operations, while it should be avoided for small writes. POSIX system calls and independent MPI-IO functions, on the other, presents a more stable performance, although higher average latency.
Although being part of an ongoing research work, initial results obtained with IORE are promising and indicate its potential contribution to the field. In short-term, we intend at including new features, such as support for particle datasets, and new AFIO implementations, such as HDF5, and NETCDF. Later, we intend to investigate other types of I/O workloads, such as those observed in Big Data and Internet of Things applications, and how they could be reproduced using IORE.
Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). We also would like to thank INESC P&D Brasil support.