In 2018, the “Govorun” supercomputer was put into operation at the Laboratory of Information Technologies and is currently used to solve a wide range of tasks facing the Joint Institute for Nuclear Research (JINR). The “Govorun” supercomputer is a heterogeneous computing environment that contains Intel CPUs of different types and NVIDIA Tesla V100 graphics accelerators. The heterogeneous structure of the supercomputer allows users to choose optimal computing architectures for solving their tasks. One of the features of modern scientific tasks, both in the field of theoretical studies and tasks related to experimental data processing, is the analysis of large amounts of data. To accelerate the processing of big arrays of data, a hierarchical hyper-converged data processing and storage system with a software-defined architecture was implemented on the “Govorun” supercomputer. According to the speed of accessing data, the system is divided into layers that are available for the user's choice. Each layer of the developed data storage system can be used both independently and as part of data processing workflows. It is noteworthy that a part of the cold storage is managed by the geographically distributed EOS file system, which allows one to connect the data processing and storage system implemented on the “Govorun” supercomputer to geographically distributed storages, the so-called DataLakes. The implemented hierarchical data processing and storage system provides the low time of data access and a data read/write speed of 300 Gb/s. The heterogeneous structure of the supercomputer and the implemented hierarchical data processing and storage system enables the cardinal speed-up of research underway at the Institute. The article describes some examples of using the resources of the “Govorun” supercomputer. The results of studies conducted by different scientific groups using the resources of the supercomputer were published in more than 70 world's leading scientific journals. The studies in this direction were supported by the RFBR special grant (“Megascience - NICA”), No.1802-40101.
The HybriLIT heterogeneous platform [1] is a component of the Multifunctional Information and Computing Complex of the Laboratory of Information Technologies of the Joint Institute for Nuclear Research (LIT JINR) [2]. The Platform consists of the education and testing polygon and the “Govorun” supercomputer, combined by a unified software and information environment. The major purpose of the Platform is to cardinally accelerate theoretical and experimental studies underway at JINR in the field of elementary particle physics, nuclear physics and condensed matter physics, including for the implementation of the scientific program at the NICA (Nuclotron-based Ion Collider fAcility) accelerator complex [3]. In addition, the resources of the Platform are used to develop parallel applications, information systems and environments for solving applied tasks and training specialists in the field of high-performance computing and modern methods and algorithms for Big Data analytics.
The education and testing polygon, which mainly implements training programs and the development of parallel applications, including hybrid ones, consists of four nodes with NVIDIA Tesla K80 graphics processors, four nodes with NVIDIA Tesla K40 accelerators, one node with Intel Xeon Phi 7120P coprocessors, as well as one node with two types of computing accelerators NVIDIA Tesla K20x and Intel Xeon Phi 5110P. All the nodes have two Intel Xeon multi-core processors. Overall, the cluster contains 252 CPU cores, 77,184 GPU cores, 182 PHI cores, 2.4 TB RAM and 57.6 TB HDD, and has a total performance of 142 Tflops for single-precision operations and 50 TFlops for double-precision operations.
The “Govorun” supercomputer, named after the Corresponding Member of the USSR RAS N.N. Govorun, one of the founders of LIT JINR, is aimed to perform resource-intensive, massively parallel calculations. The “Govorun” supercomputer is an innovative hyper-converged software-defined system and has unique properties for the flexibility of customizing the user’s job, ensuring the most efficient use of the computing resources of the supercomputer. The “Govorun” supercomputer comprises a GPU component, a CPU component and a hierarchical data processing and storage system. The GPU component is implemented on the basis of five DGX-1 servers from NVIDIA, each of which contains eight Tesla V100 GPUs, while the servers are connected by a high-speed InfiniBand network with a bandwidth of 100 Gb/s. The CPU component of the supercomputer is implemented on the “RSC Tornado” high-density architecture with direct liquid cooling, which ensures a high density of compute nodes, i.e. 150 nodes per rack, and high energy efficiency about 10 GFlop/WВ [4]. The average annual PUE indicator of the system, reflecting the level of energy efficiency, is less than 1.06. It means that less than 6% of all electricity consumed is spent on cooling, which is an outstanding result for the HPC industry. At the same time, it is noteworthy that the transition to liquid cooling is a global HPC trend. The CPU component includes 88 nodes based on Intel Cascade Lake processors with Intel® SSD DC P4511 high-speed solid-state drives and the 2 TB NVMe interface, as well as 21 nodes based on Intel Xeon Phi processors. The total peak performance of the supercomputer is 860 Tflops for double-precision operations and 1.7 Pflops for single-precision operations. The CPU component of the “Govorun” supercomputer is ranked 11th, and the GPU component is 21st in the Top50 list of the most powerful supercomputers in Russia and the CIS [5].
To accelerate work with data, a hierarchical hyper-converged data processing and storage system with a software-defined architecture was implemented on the “Govorun” supercomputer. The hyperconverged architecture of the supercomputer enables the creation of a high-speed data storage and processing system with a parallel file system speed of about 300 Gb/s for reading/writing, which is an extremely convenient tool for processing big arrays of data, including for the NICA megaproject. At present, the “Govorun” supercomputer is ranked on the 22nd place in the IO500 list (November 2020) [6]. The compute nodes of the CPU component, as well as the connection between the GPU component and the hierarchical data processing and storage system, are implemented on the basis of a high-performance Intel Omni-Path network with a bandwidth of 100 Gb/s.
A wide range of tasks both in the field of theoretical studies and in the field of experimental data processing, related to the JINR scientific program, is solved on the “Govorun” supercomputer. One of the major tasks solved on the supercomputer, involving the CPU and GPU components, as well as the data storage system, is to carry out calculations within lattice quantum chromodynamics. At present, methods and algorithms of machine and deep learning are actively developed and implemented for radiation biology tasks, and the neural network approach is developed for track detection and reconstruction for the experiments of the NICA megascience project and the neutrino program. One of the sources of resourceintensive tasks is the MPD experiment of the NICA project, for which the tasks of event generation and reconstruction are solved. The effective solution to such a wide range of heterogeneous tasks entails the development and support of a flexible software and information environment that satisfies requests from different user groups for both the installation and updating of application software and the development of novel IT solutions, including middleware, which allows integrating computing resources and data storage resources into a unified space.
The HybriLIT platform is an actively developing system that includes additional subsystems, apart from two main elements, i.e. the education and testing polygon and the “Govorun” supercomputer. The subsystems encompass a pool of virtual machines that provide remote access to the Platform, a pool of virtualized servers that ensure work with graphically loaded application packages, data storage servers, virtual machines and servers for the ecosystem of machine and deep learning and HPC (ML/DL/HPC ecosystem), virtual machines to support information services. Figure 1 illustrates the Platform structure.
All the elements of the Platform are integrated into a unified software and information system, which allows users to work in different modes without data transfer, program compilation, etc. For example, users can develop and debug parallel applications on the education and testing polygon, perform resource-intensive calculations on the “Govorun” supercomputer, and analyze and visualize the results within the ML/DL/HPC ecosystem.
The software and information environment can be divided into two levels, i.e. the level of information and user support and the level of system software and IT services.
The first level comprises the HybriLIT web site (http://hlit.jinr.ru), which contains detailed information about the resources provided to users, software, instructions for working on the Platform, including video materials. GitLab (http://gitlab-hybrilit.jinr.ru) is supported for efficient organization of user work, for joint parallel development of applications, as well as for communication with the group of system administrators of the Platform. The Indico system (http://indico-hybrilit.jinr.ru) is supported for organizing conferences, seminars and meetings devoted to parallel programming technologies. One can also download materials of lectures and seminars in the system, which allows users to get acquainted with them in more detail.
The second level includes the basic software, i.e. the Scientific Linux operating system, the SLURM package-scheduler responsible for distributing the flow of jobs between the Platform nodes and a set of basic compilers. The CernVM File System (CernVM-FS) [7] is used as a means of user access to software on the HybriLIT heterogeneous platform. Internally, CernVM-FS utilizes an address-oriented storage and hash trees to support file data and metadata. The CernVM-FS file system is read-only available to users, which avoids problems related to accidental rewriting of service files and libraries. CernVM-FS transfers data and metadata on demand and verifies data integrity using specialized cryptographic keys. Files and directories are hosted on standard web servers and mounted in the universal namespace /cvmfs. At the stage of calling the software located in CernVM-FS, libraries, compilers, etc., necessary for the launch and operation of the main functionality, are uploaded to the node. After the program has closed and a specific time has passed, the corresponding directories related to the previously launched application are unmounted from the node.
By caching a multitude of small read-write-intensive utility files, the performance of CernVM-FS is significantly enhanced, which accelerates user applications.
The following scheme for working with the CernVM-FS file system was implemented on the HybriLIT platform:
1. On the soft1-hlit.jinr.ru server, the necessary packages are installed, the modules for setting up user environment variables are prepared.
2. Data from soft1-hlit.jinr.ru are transferred to the cvmfs0-hlit.jinr.ru server (Stratum 0) – a central repository that stores all data on the installed software. From there, the data are synchronized with cvmfs1-hlit.jinr.ru (Stratum 1) – a replication server that increases reliability, reduces load and protects the main copy of the Stratum 0 storage from direct user access.
3. When the software is launched in the interactive mode or on the compute nodes of the Platform, the required software is loaded onto these nodes.
The Modules 3.2.10 package (http://modules.sourceforge.net/) is used to directly configure the user environment variables on the HybriLIT platform. Each module file contains information needed to set up the environment variables for the corresponding software. These files include values for initializing or changing variables such as C_INCLUDE_PATH, LIBRARY_PATH, PATH, etc. The module files can be shared by many users of the Platform. Users themselves can customize the use of each required module by editing the corresponding files in their home directory.
To increase the efficiency of solving user jobs, as well as to expand the efficiency of the utilization of both the computing resources and data storage resources, resource orchestration was implemented on the “Govorun” supercomputer. This notion means software disintegration of a compute node, i.e. the separation of compute nodes and data storage elements (SSDs) with their subsequent integration in accordance with the requirements of user jobs. The process is shown schematically in Figure 3.
Thus, the computing elements (CPU cores and graphics accelerators) and data storage elements (SSDs) form independent fields. Due to orchestration, the user can allocate for his job the required number and type of compute nodes (including the required number of graphics accelerators), the required volume and type of data storage systems. After the job is completed, the compute nodes and storage elements are returned to their corresponding fields and are ready for the next use. This feature allows one to effectively solve user tasks of different types, to enhance the level of confidentiality of working with data and avoid system errors that occur when crossing the resources for different user tasks.
It is noteworthy that modern HPC systems are used not only as traditional computing
environments for carrying out massively parallel calculations, but also as systems for Big Data analysis
and artificial intelligence tasks that arise in different scientific and applied tasks. At the same time,
despite the increase in supercomputer performance, memory and data storage bandwidths become
bottlenecks. To accelerate work with data for tasks of different types, solved on the “Govorun”
supercomputer, a hierarchical hyper-converged data processing and storage system with a
softwaredefined architecture was developed and implemented. It realizes a new paradigm for working with data,
namely, the integration of computing elements and novel types of data storage elements (Intel Optane,
Intel SSD) [8] into a unified computing environment. According to the speed of accessing data, the
system is divided into levels that are available for the user’s choice, namely, a super-hot layer
implemented on the basis of Intel Optane, a hot layer based on Intel SSD NVMe under the management
of the Lustre file system, a warm layer implemented as “an on-demand storage system”, which can be
managed by different file systems defined by the user, a cold layer implemented on HDD of sufficient
volume, which ensures data storage, but does not meet the peak requirements of a computational task.
Each layer of the data storage system under development can be used both independently and as part of
data processing workflows. It is noteworthy that a part of the cold storage is managed by the
geographically distributed EOS file system [9], which allows one to connect the data processing and
storage system implemented on the “Govorun” supercomputer to geographically distributed storages, the
so-called DataLakes. The super-cold layer is a tape storage. The DIRAC software [
To support the development of methods and algorithms of machine and deep learning, as well as the elaboration of an environment for data analysis and visualization, an “Ecosystem for the tasks of machine learning, deep learning and data analysis” was introduced into the software and information environment of the HybriLIT platform. The creation of the ecosystem is related to the active implementation of the neural network approach, ML/DL methods and algorithms for solving a wide range of tasks, which is defined by many factors. The development of computing architectures, especially while using DL methods for training convolutional neural networks, the development of libraries, in which various algorithms are implemented, and frameworks, which allow building different models of neural networks, can be referred to the main factors. To provide all the possibilities both for developing mathematical models and algorithms and carrying out resource-intensive computations, including graphics accelerators, which significantly reduce the calculation time, the ecosystem with three components was introduced (Fig.5).
The ecosystem is implemented on the JupyterHub basis, i.e. a multi-user platform for working with Jupyter Notebook (known as IPython with the possibility to work in a web browser), which comprises a set of libraries and frameworks:
• computational component designed to perform resource-intensive, massively parallel tasks of training neural networks using NVIDIA graphics accelerators (https://jhub2.jinr.ru);
• JLabHPC component enabling calculations on the compute nodes of the HybriLIT platform, including on the “Govorun” supercomputer (https://jlabhpc.jinr.ru); • component for the development of models and algorithms and data analysis (https://jhub.jinr.ru).
The resources of the “Govorun” supercomputer are used by scientific groups from all the Laboratories of the Institute within 25 themes of the JINR Topical Plan for solving a wide range of tasks in the field of theoretical physics, as well as for modeling and processing experimental data.
One of the most resource-intensive tasks involving all the computing components and the data storage system of the “Govorun” supercomputer is to perform calculations for studying the properties of quantum chromodynamics (QCD) and Dirac semimetals in a tight-binding mode under extreme external conditions using lattice modeling. The given study entails the inversion of large matrices, which is carried out on graphics accelerators, as well as massive parallel CPU computations, to implement the quantum Monte-Carlo method. Within this direction, the following tasks were solved (Fig.6):
– the influence of the magnetic field on the confinement/deconfinement transition and the chiral transition at finite temperature and zero baryon density were investigated using the numerical modeling of lattice QCD with a physical quark mass.
– quantum chromodynamics with non-zero isospin density taking into account dynamical u- d-, squarks in the Kogut-Susskind formulation was studied.
– the potential of the interaction between a static quark-antiquark pair in dense two-color QCD was investigated, and the confinement/deconfinement phenomenon was studied.
– the effect of the non-zero chiral chemical potential on dynamical chiral symmetry breaking for Dirac semimetals was studied.
– the influence of the external magnetic field on the electromagnetic conductivity of quark-gluon plasma was investigated.
The calculations for the study of Compton scattering at helium atoms, conducted by an international research group, which comprises theorists from different countries and experimenters from Goethe University (Frankfurt am Main, Germany), may serve as another example of resource-intensive tasks. A kinematically complete measurement of the characteristics of Compton scattering at free atoms was performed using a highly efficient method called COLd Target Recoil Ion Momentum Spectroscopy (COLTRIMS), and its relevant theoretical description was provided. To do this, the experimenters directed a powerful photon beam of the Petra III synchrotron (DESY, Hamburg) through a supersonic helium flow. The COLTRIMS method allowed one to measure not only the momentum of a scattered electron, but also the recoil momentum of a helium ion for individual scattering events, which, taking into account the law of conservation of energy-momentum, made it possible to completely restore the kinematic characteristics of the scattering process. In addition, the use of this method solved the problem of a very small cross section of Compton ionization in the photon energy range of an order of several keV, which is about six orders of magnitude lower than the typical photoabsorption cross section. It opens up possibilities for the use of Compton scattering as another tool of atomic spectroscopy, along with such powerful methods of studying atoms and molecules as (e, 2e), (ion, ion e), etc. (Fig.7)
The tasks of data mass generation and reconstruction for the NICA MPD experiment may serve as an example of extramassive calculations that actively utilize the hierarchical data processing and storage system of the “Govorun” supercomputer. The experience of using different computing resources of JINR and other institutes of the MPD collaboration has shown that at the moment the use of the computing resources of the “Govorun” supercomputer is the most efficient. The use of only 270 compute cores (such a limited resource is related to the insufficient resources and high load of the supercomputer) provides data processing equivalent to the use of 450-500 cores on the other available computing resources, such as Tier1, Tier2 and the NICA computing cluster of the Laboratory of High Energy Physics (VBLHEP). On the computing resources of JINR and the National Autonomous University of Mexico, over 50 million events were modeled and processed by the MPD collaboration for 2020, and a quarter of these events were performed directly on the “Govorun” supercomputer. The unique equipment of the “Govorun” supercomputer, which comprises the hierarchical data processing and storage system, made it possible to process the same number of events on almost half the number of compute cores as on the other available computing resources (Fig.8).
Fig.8. Computing resources used for the NICA MPD experiment under the management of the DIRAC software
Using the ML/DL/HPC ecosystem, algorithms that allow solving the tracking task for the BM@N experiment, i.e. combining individual hits (traces of the manifestation of charged particles) into a group corresponding to each particle separately, were developed. Such a task is well known in high-energy physics, and with the development and, accordingly, complication of both experiments and experimental facilities themselves, as well as with an increase in the energy and mass of colliding particles, the complexity of this task significantly grows. The global detection of tracks among noises is performed immediately over the entire picture of the event. The GraphNet program under development is based on the use of graph neural networks for tracking. An event is represented as a graph with counts as nodes, then this graph is inverted into a linear orggraph, when the edges are represented by nodes, and the nodes of the original graph are represented by edges. In this case, the information about the curvature of track segments is embedded in the edges of the graph, which simplifies the detection of tracks among fakes and noises.
The black nodes and the edges correspond to fakes, the green nodes and the yellow edges correspond to found tracks.
The results of the calculations on the “Govorun” supercomputer allow one to estimate the rate of processing one event of the future detector HL-LHC or NICA with 10,000 tracks at a reasonable level of three microseconds [21-22].
Another example of using the ML/DL/HPC ecosystem is the development of an information system for the tasks of radiation biology, which is aimed at storing experimental data and analyzing changes in the central nervous system of mammals on the basis of molecular, pathomorphological and behavioral changes in the mammalian brain when exposed to ionizing radiation and other factors (Fig.10).
The heterogeneity of experimental data in this area of research defines the creation of a developed subsystem for acquiring, storing and systematizing experimental data, which is capable of working with data with the same efficiency for both conducted and current experiments when studying the effects of ionizing radiation and other factors on biological objects. At the same time, a complete picture and a model of the ongoing processes require the development of a qualitative set of algorithms for experimental data processing based on machine and deep learning methods. Close interaction of different research groups and the requirement to ensure information security when accessing data and research results entail the application of a maximum of modern IT solutions, including web technologies, reliable modern means of authentication and hierarchical access management, as well as components for convenient operation and visualization of data analysis results [23].
The operation of the JINR supercomputer named after N.N. Govorun in 2018-2020 made it possible to perform a number of complex resource-intensive calculations in the field of lattice quantum chromodynamics to study the properties of hadronic matter at high energy density and baryon charge and in the presence of supramaximal electromagnetic fields, to qualitatively increase the efficiency of modeling the dynamics of collisions of relativistic heavy ions, to speed up the process of event generation and reconstruction for conducting experiments within the NICA megaproject implementation, to carry out computations of the radiation safety of JINR experimental facilities, to significantly accelerate studies in the field of radiation biology and other applied tasks solved at JINR at the level of international scientific cooperation. The results of the studies were published in more than 70 world's leading scientific journals.
An equally important aspect of the activity that involves the resources of the HybriLIT platform is the educational direction related to both training courses for JINR employees and practical classes for students of Dubna State University, Tver State University and other universities [24]. The Platform resources are also actively used to train IT specialists within the International School of Information Technologies “Data Science”, whose students are engaged in real scientific projects of JINR [25]. Three PhD theses and more than 60 master’s and bachelor’s theses were prepared using the resources of the HybriLIT platform.
In conclusion, it should be noted that the HybriLIT platform, which comprises the “Govorun” supercomputer and the education and testing polygon, is an actively developing environment combining modern computing architectures and IT solutions, which enables the cardinal acceleration of research underway at JINR, and providing a basis for the future human resources of the Institute in information technology. (accessed on: TOP50 rating [http://top50.supercomputers.ru], rating of CIS supercomputers. Edition No.32 of 31.03.2020.
Semin А. DAOS: Data storage systems for HPC/BigData/AI applications in the era of exascale computing, Storage News, No.2 (74), 2019, in Russian.
[19] Moshkin A., Rogachevsky O., Pelevanyuk I. Centralized Monte-Carlo productions Electronic resource:
https://indico.jinr.ru/event/1469/contributions/9914/attachments/8222/12277/mpd-comp.pdf
Korenkov V.V., Podgainy D.V., Streltsova O.I. Educational program on HPC technologies based on the HybriLIT heterogeneous cluster (LIT JINR). International scientific journal “Modern information technology and IT education”, v. 13, no. 4, p. 141-146, dec. 2017. ISSN 2411-1473, in Russian.