Since 2019, the COMPASS experiment has been running on the Frontera high-performance computer. This is a large machine (number 5 in the ranking of the most powerful supercomputers in 2019). Details of software setup and approaches to organizing data processing on this machine are presented in the article.
The COMPASS experiment [
Frontera has two computing subsystems: a primary computing system focused on
doubleprecision performance and a secondary subsystem focused on single-precision streaming memory
computing. Frontera also has several storage systems, interfaces to the cloud and archive systems, and
a set of nodes for hosting virtual servers. The primary computing system is Dell EMC on Intel Xeon
Platinum 8280 (“Cascade Lake”) processors, 56 CPUs per node, 192GB of RAM per node, linked by
the Mellanox Infiniband HDR and HDR-100 interconnect. The system consists of 8,008 compute
nodes. Slurm [
Data processing on high-performance computing systems (High Performance Computer, HPC) has several major differences compared to working on ordinary clusters, which are traditionally used in high-energy physics: these are conceptually different systems, but the relative similarity of the organization of computational nodes and system software makes it possible to run the applied software of experiments on supercomputers without changing the source code. The differences are architecturedriven and must be taken into account when organizing processing. They can be presented as the following list:
• each supercomputer is a machine built in a single copy and has unique characteristics inherent only to it;
• two-factor authentication is a common type of authentication on an HPC, which reduces the availability of a resource for automatic data processing systems – it is not always possible to access a computing resource from the outside, as is usually done when working with the nodes of a grid environment – it is necessary to place services, responsible for communication with the workload management system, on login or service nodes on an HPC;
• a shared file system requires delicate I/O management so as not to overload the file system and interfere with the work of other users;
• cluster management system: each HPC has its installation of one of the versions of a specially adapted scheduler and resource manager, for example, PBS, PBS Pro or Slurm; • specialized system software; • inability to use grid middleware such as CVMFS clients, VOMS, data management tools; • lack of access to the Internet at compute nodes; • user and project policy usually requires strict use of a computing resource for a limited period of time;
• usually the time for storing data on local storage is limited and not available throughout the project.
From the above, it follows that to organize data processing on such kinds of computing systems, it is necessary to carry out a deep modernization of all components of the computing infrastructure of the experiment.
The PanDA workload management system manages the full cycle of data reconstruction jobs on Frontera, including both processing and merging intermediate results into the final ones [fig. 1].
Unlike the infrastructure implemented for Blue Waters, Frontera uses a new task management
service named Harvester [
Harvester is a component of the PanDA ecosystem responsible for providing communication
between WMS and pilots working on compute nodes. It can be located both on a dedicated node in the
case of working with computing resources on the grid, and on a service node of high-performance
computing systems. Harvester is a multithreaded service that uses an internal database to store all the
necessary information. It can manage not one submission on a high-performance system, but a set,
including various sizes and execution times. On supercomputers, using Harvester, it is possible to
organize the processing of a multitude of large jobs. With the advent of this service, the issue of
organizing processing on HPC systems has been completely resolved: for example, the ATLAS
collaboration has fully integrated all available high-performance systems from the USA and
Europe.[
Several modules were added to the code of Harvester to control the processing of COMPASS data. Everything needed for the COMPASS job code, such as local MySQL database execution, payload management, errors handling and stage-out, was transferred from the Multi-Job Pilot and added as Harvester plug-ins. Since Frontera is one of the most powerful computing systems in the world, much attention is paid to optimizing I/O operations – when performing a large number of jobs there is a risk of overloading the metadata server of the shared file system. For this reason, all the software needed to complete the job is moved to a temporary folder on the compute node. The database server used by the tasks, which stores the calibration information, is also copied there. All data transfer operations to and from the node are monitored to avoid overloading the file system.
The software infrastructure on Frontera includes 4 Harvester services, each of which is configured to perform 5 supercomputer tasks with 50 compute nodes each. In terms of grid, this is 56,000 individual jobs [fig. 2]. To handle such load, it was necessary to deploy an additional PanDA server, reserved only for working with Frontera. It was also required to increase the capacity of the database server storing information about jobs. The size of the jobs is due to the fact that on Frontera larger jobs are given priority over smaller jobs. The size of the jobs is selected in such a way that, on the one hand, it is fast enough to get access to resources, and on the other hand, so as not to overload the IT infrastructure of the experiment that controls data processing.
Since the entire chain of data reconstruction is performed on Frontera, including reconstruction jobs, as well as merging jobs of each result type: mDST, histograms, and files of selected events, Harvester was tuned for optimal performance. The fact is that reconstruction jobs are usually completed within 7-8 hours, while merging jobs are completed within an hour. To organize such work, two queues were registered in PanDA: for reconstruction jobs and for merging jobs. In Harvester, each queue has its own settings: the queue for reconstruction jobs requests 50 nodes for 10 hours, and for merging 1 node for 2 hours. Exceeding the requested time is done to prepare the input data, to have time to stage-out the output and ensure the reliable execution of jobs in the event of possible slowdowns in the work of the shared file system.
Usually, when working on large computing systems, priority is given to larger jobs. In the case of Frontera, jobs of 50 nodes are considered as small and can spend up to a week waiting in the queue before reaching compute nodes. Such behavior is inconsistent with the concept of highthroughput computing, in which all components of the distributed computing infrastructure, such as PanDA, were developed. In the grid, strong connectivity with advanced monitoring tools is practiced: each job reports its status every 30 minutes. This is done to minimize the wait time of computing resources and ensure a stable number of jobs performed. When waiting in a queue for a week, such high connectivity becomes not only unnecessary, it becomes a generator of excess load on all components of the computing infrastructure.
The PanDA server is restarted once a day to reset the logger handlers and download information about the certificate revocation list. However, even the fastest server reboot under high load results in the loss of monitoring packets directed by the Harvester services to the PanDA server. The omission of the message about the status of the job leads to the recognition of the jobs as lost: the PanDA server sends a message to Harvester to remove such a job from the queue. In a grid environment, each job is processed individually, and the loss of several jobs does not lead to noticeable consequences. In the case of high-performance systems, deleting one job results in the deletion of the entire submission of 2,800 jobs, which is completely unacceptable.
To eliminate this behavior, the PanDA server was reconfigured. The logging level was set to CRITICAL, the fetch-crl service was disabled, and server reboot was disabled. A comparative profile of I/O operations before and after the reconfiguration is shown in Figure 3. Reconfiguring the PanDA server resolved the missed packets issue. Now the PanDA server can work for several months without restarting.
Thus, the use of high-performance systems places increased demands on all components of the data processing infrastructure. This is due to the abrupt load profile characteristic of computing on high-performance systems: in the case of working on a grid farm, jobs are delivered to computational nodes and launched one at a time, while in the case of working with a supercomputer, several thousand jobs are grouped into one submission and, after waiting in the queue, start almost simultaneously. In addition, submissions for a larger number of nodes on such systems receive a higher priority in the queue, so it is necessary to keep as many large submissions in the queue as possible. A new implementation of the pilot application was developed to work on Frontera. A diagram of the data processing organization on the Frontera HPC is shown in Figure 4.
Submissions for 50 computational nodes, each of which consists of 2,800 individual jobs,
were used for processing. The supercomputer was utilized to carry out the entire chain of data
reconstruction tasks processing, including the aggregation of results. The management of tasks and
jobs on this resource was fully integrated into the production system of the COMPASS experiment
and was performed in the same way as data processing management on grid clusters [
The study was supported by the Russian Science Foundation grant (project No. 19-71-30008). Workshop