At present, the “Govorun” supercomputer is used for both theoretical studies and event simulation for the MPD experiment of the NICA megaproject. To generate simulated data of the MPD experiment, the computing components of the “Govorun” supercomputer, i.e. Skylake (2880 computing cores) and KNL (6048 computing cores), are used; data are stored on the ultrafast data storage system (UDSS) under the management of the Lustre file system with a subsequent transfer to cold storages controlled by the EOS and ZFS file systems. UDSS currently has five storage servers with 12 SSD disks using the NVMe connection technology and a total capacity of 120 TB, which ensures a low time of data access and a data acquisition/output rate of 30 GB per second. Due to the UDSS high performance, by September 2019, over 100 million events for the MPD experiment have already been generated and more than 30 million events have been reconstructed. In the future, other MC generators are expected to be used as well. It is planned to use the DIRAC software for managing jobs and the process of reading out/recording/processing data from various types of storages and file systems. All the enumerated above will allow one to check a basic set of data storage and transmission technologies, simulate data flows, choose optimal distributed file systems and increase the efficiency of event modeling and processing.
The Heterogeneous platform “HybriLIT” [
The “Govorun” supercomputer [
To speed up work with data, an ultrafast data storage system (UDSS) is implemented in the
“Govorun” supercomputer under the Lustre file system. At present, UDSS has five storage servers
with 12 SSD drives with the NVMe connection technology, which reduces the time of access to data.
The total capacity of UDSS is currently 120 TB and the data acquisition/output rate is 30 TB per
second. It is noteworthy that UDSS has the ability to linearly increase productivity (speed of working
with data) and the storage volume without changing the principles of the architectural design of the
system. It should also be highlighted that a new approach for the industry of high-performance
systems, based on the principle of hyper-convergence of the system, the essence of which is to
combine resources for computing and storage on each node of the system, is implemented in the
“Govorun” supercomputer. Each node of the system is both part of the computing subsystem and part
of the distributed system of user data storage, performing two types of load at once (compute/store). It
allows linearly scaling the system resources with an increase in the number of nodes. Unlike the
classical approach in HPC, when the computing system and the storage system are separate and scale
separately, in the hyper-converged system, as the number of nodes increases, both the computing
power and the volume/speed of the distributed data storage system grow in parallel [
Thus, the uniqueness and advantage of the CPU component of the given supercomputer are 100% liquid cooling, which ensures a high density of computing nodes, i.e. 150 per cabinet, and high energy efficiency, about 10 GFlop/W; hyper-convergence and the ultrafast data storage system enabling to significantly accelerate data processing and being unique in the field of high-performance computing; heterogeneity, which allows solving tasks that require different computing architectures.
The total peak supercomputer performance is 1 PFlops for single-precision operations and 500 TFlops for double-precision operations.
At present, the computing resources of the “Govorun” supercomputer are actively used for
both theoretical studies in lattice quantum chromodynamics and event generation and reconstruction
for the NICA experiments. For theoretical research, the GPU and CPU components are used, while for
event generation and reconstruction, only the CPU components, namely, Skylake and KNL, are used;
data are stored on UDSS with a subsequent transfer to cold storages controlled by the EOS and ZFS
file systems. The DIRAC software [
Using DIRAC, the computing resources of the JINR MICC, i.e. Tier1/Tier2, the “Govorun” supercomputer, the JINR cloud and storage resources such as Lustre UDSS, dCache and EOS, were combined. Within Monte-Carlo data generation for the MPD experiment, 70,000 jobs were performed on the MICC Tier1/Tier2 components using the DIRAC platform, and 15,000 jobs were carried out on the computing resources of the “Govorun” supercomputer using UDSS. As a result, 4.5 TB of data were generated and sent to dCache.
In total, by September 2019, over 167000 tasks on all computing components were completed by all groups performing calculations on the supercomputer. It is noteworthy that about 31% of them refer to computing for the NICA megaproject. Moreover, more than half are the theoretical calculations carried out on all computing components of the supercomputer. More than 40% directly relate to event generation and reconstruction for the MPD experiment, namly, 75 million events have been generated using UrQMD, LAQGSM, PHSD and other models and 30 million events have been reconstructed (Fig.3).
One of the major tasks, for which the resources of the supercomputer will be used, is the development of computing for the NICA megaproject. The created computing for the NICA megaproject should provide data acquisition from detectors and data transmission for processing and analysis. To perform the given tasks, computing has certain requirements, including the requirements to the network infrastructure, computing architectures, storage systems, as well as the appropriate software of the system and for data processing and analysis. Developed computing models should take into account the trends in the development of network solutions, computing architectures and IT solutions, which allow combining supercomputer (heterogeneous), grid and cloud technologies and creating distributed, software-configured HPC platforms on its basis. The use of such solutions for data processing and analysis requires the creation of software environments, which provide the necessary code abstraction enabling to implement the required functionality for a wide range of computing tools.
The implementation of various computing models for the NICA megaproject entails the confirmation of the model performance, i.e. meeting the requirements for temporal characteristics of data acquisition from detectors with their subsequent transfer to processing, analysis and storage, as well as the requirements for the efficiency of modeling and processing events in the experiment. For these purposes, it is necessary to carry out tests in a real software and hardware environment, which should contain all the required components. Supercomputer "Govorun" is the hyperconverged computational system and contains all newest computational architecture and ultrafast data storage system and that why it is one of the major elements for performing a full cycle of testing a computing model for NICA megaproject.
Acknowledgments
The studies in the given direction were supported by the RFBR grant (“Megascience – NICA”), №18-02-40101.