The ROOT software package is currently being upgraded in several ways to improve data processing performance. This paper will consider several tools implemented in this framework for calculations on modern heterogeneous computing architectures. PROOF (Parallel ROOT Package Extension) divides common work into small chunks, i.e. packets. The size of the first packet used for calibration, the minimum and maximum set size of the packet, the degree of data structuring affect the speed of their processing. When processing large amounts of data, the read and write speed can be crucial. The new asynchronous file merge feature in the TBufferMerger class allows writing data in parallel from multiple streams to a single output file. Our calculations show a good scalability of the macro execution time depending on the number of processor cores used.
The ROOT [
PROOF [
The calculations were performed using the ROOT 6.13 and PROOF-Lite versions. For their operation on the HybriLIT cluster, six CPU cores were reserved. PROOF was running at different values of the number of workers. As long as the number of workers started does not exceed the number of cores, one worker process completely occupied the processor core. If, when starting PROOF, more workers than the number of reserved cores were called, then naturally; several workers were sharing one processor core. The acceleration factor (the ratio of the execution time of a macro in a non-parallel version and the time of a macro in a parallel version) was actively growing with an increase in workers from two to six, then its value stabilized. When starting PROOF with more than 12 workers, the acceleration rate decreased smoothly. The data shown in the graphs was obtained with workers equal to six.
To represent data in ROOT, a special class TTree was developed, it was optimized to reduce disk space and increase the data access rate. All variables are presented in the form of leaves, which are combined into branches, and the collection of branches forms a tree. The tree storage approach is good for parallel architectures since each branch can be read independently of the other. The tree can have different structures. We were interested in the question of how the organization of data in a tree affected the effectiveness of using PROOF. We considered a simple tree with a list of variables, a tree with structures and a tree with class objects.
All of our trees contained twelve variables structured in different ways. The simple tree had twelve branches, where a floating-point variable acted as a leaf. In the second case, the tree had three branches, the first two of which contained three leaves each, combined into structures, and the third branch was a structure of six float variables. In the third case, the tree had one branch that contained an object consisting of all twelve variables. The data was contained in ten files, each of which was approximately 1.6 GB in size. We performed two analyzes. In the first of them, data was read from all branches, they were processed and the result was written to a new file. In the second analysis, the data of only two variables was read, and in the case of the tree with a structure, we considered two options for reading, namely, in the first, two variables belonging to the same structure were read and processed, and in the second case, the variables belonged to different structures. The processing included calculations of various physical quantities commonly used in different types of physical analysis.
As seen from the graphs, when reading and processing all data using PROOF, the acceleration factor has a maximum value if the data is organized into structures. If only individual columns of root files are read and processed, then the maximum speedup factor is achieved when organizing the data in the form of a tree containing objects.
Different types of analysis have different ratios of time over which different processing steps
are performed. When processing large volumes, sometimes the time spent on reading or writing data
comes to the fore. New possibilities for parallelizing these processes are actively used in the
optimization of HEP software [
A distinctive feature of ROOT, which made it easy to use, is its columnar data format. The user has the ability to read the data of only those columns that are of interest to him. In the process of reading, the data is unpacked and deserialized. Starting from version 6.08, functions that perform these actions in parallel have been implemented in ROOT. The user only has to specify the number of threads. All implementation details are hidden from the user, which is why this approach is called Implicit Multithreading.
For parallel writing, the TBufferMerger class, implemented in ROOT since version 6.10, is used. This class implements the following scheme of actions. The user specifies how many streams that write data to a single file to create. The generated worker threads divide the data into separate buffers, in each of which the data is serialized and compressed. Therefore, these processes occur independently in each buffer, so they can be performed in parallel. The buffers are merged before closing the output file. In the original version, the merge had a separate output stream. Then a scheme , in which the merge is done by the worker threads themselves on demand, was implemented. This made it possible to reduce the required computing resources.
Our performance tests consisted of generating data, organizing it into a tree with twelve branches, and then writing it to a file. Figure 2 on the left shows the dependence of the running time of a macro on the number of worker threads and the number of generated events. The dependence of the acceleration coefficient on the same factors is illustrated on the right.
From the results presented in the graphs, it can be seen that the processing time decreases with an increase in the number of created threads. When the number of worker threads exceeds the number of cores in use, the reduction in macro runtime stops. The acceleration factor is proportional to the number of available processor cores and does not depend on the number of generated events.
Currently, parallel computing is the main reserve for increasing the speed of calculations, therefore, the issues of parallelization of the ROOT package are of great importance. Modern architectures of computing systems make it possible to use various methods of parallelizing the processing of experimental data. The choice of the optimal method in each specific case significantly reduces the task execution time. Our research has shown that when processing data using PROOF, it is desirable to use the highest possible structuring of primary data. When working with the PROOF-Lite version, it is reasonable to set the number of workers equal to the number of reserved cores for that version. Writing to a file in parallel is easy to implement, and the results depend on the number of cores used.
The author is grateful to his colleagues for a fruitful discussion of the issues, as well as to the HybriLIT group of MLIT JINR for the resources and technical support provided.