Introduction

The most important issue in the operation of a high-performance computing cluster is to provide the complete utilization of its resources. This is necessary for solving scientific problems and ensuring the return of investments (ROI).

We can distinguish two main areas in this problem [1,2]:

-ensure execution of the maximum possible number of applications for a certain period of time; -the most efficient use of cluster resources by user applications. An important issue of operations is to determine the grade of loading of the cluster, because it allows to plan the provision of resources, to assess the necessity for modernization, to determine the quality of the services.

As a rule, the workload is defined as the ratio of the metric (parameter) of the workload to the maximum possible value of this parameter. The metric is determined by measurement or calculation.

The article proposes a new method for calculating the value of the workload using the peak performance of the cluster.

A high workload of the HPC cluster does not mean efficient use of its resources. It is possible that the resources requested by the application are not used and are idle. In this case, the workload factor of the cluster can be high, but the quality of the tasks is low.

To provide an advantage to applications that efficiently use the resources of the cluster, the article discusses a system of dynamic priorities. The system is based on determining the coefficient of profiling and using it to change the priorities of applications.

Technique for Estimating the Workload of the Hybrid HPC Cluster

For traditional supercomputers, the workload parameter may be the number of core-hours that were provided to the application for performing calculations [3]. The ratio of allocated core-hours to the maximum possible is an indicator of the cluster workload. These parameters are calculated for a certain period of time and are an integral indicator of the workload over a given period.

For hybrid architectures, this approach is less significant, since there are various types of cores in the hybrid HPC cluster, and applications reserve the calculator's resources not by the cores, but by entire graphics accelerators.

The proposed method allows to take into account this feature. The workload estimate is determined by comparing the requested by the applications and the maximum possible number of floating-point operations per period of time.

Note that there is a difference between the theoretically possible performance of cluster (peak performance) and practically achievable results. Results are determined by different tests and vary greatly depending on the type of tasks and configuration of the cluster [4].

To estimate the workload of a hybrid high-performance computing cluster, we use peak performance. It is defined as the sum of the peak productivities of its componentsnodes (1).

𝑃 𝑝𝑒𝑎𝑘 = ∑ 𝑃 ℎ𝑜𝑠𝑡 𝑖 𝐾 𝑖=1 , (1)

where Ppeak is peak performance of the computing cluster, Phost ipeak performance (Phost) of the i-th node of the computing cluster Note that the summation does not take into account the performance losses that occur when the nodes interact over the computer network connecting them (interconnect) [4].

The peak performance of the P host node is defined as the sum of the performance of the central processors of the node (P cpu ) and its graphic accelerators -P gpu . It is assumed that they are fully loaded with floating point operations, do not perform any other operations, and there are no data transfer losses between the central processors and graphics accelerators (2).

𝑃 ℎ𝑜𝑠𝑡 = 𝑁 𝑐𝑝𝑢 𝑃 𝑐𝑝𝑢 + 𝑁 𝑔𝑝𝑢 𝑃 𝑔𝑝𝑢 , (2)

where Ncpu is number of CPUs in the compute node, Ngpunumber of graphics accelerators in the compute node, Pcpupeak CPU performance, Pgpugraphics accelerator peak performance.

To calculate the peak performance of the CPU (3), we assume that the operations are performed by the cores in parallel, each core can process a group of threads, and the flow allows several operations to be performed in parallel if there are several operational blocks for this. Such a core-streaming architecture is characteristic of modern classical processors of various manufacturers.

𝑃 𝑐𝑝𝑢 = 𝑛 𝑐𝑜𝑟𝑒 𝑛 𝑠𝑡𝑟𝑒𝑎𝑚 𝑛 𝑢𝑛𝑖𝑡 𝐹 𝑐𝑝𝑢 , (3) where ncore is number of CPU cores, nstreamthe number of threads processed by the CPU core, nunitthe number of operating units per flow corresponds to the number of operations performed in one flow per cycle, Fcpu -CPU frequency.

To assess the performance of graphics accelerators, we use the the modern accelerator architecture of the NVidia company. Consider the family of accelerators Tesla Volta, as the most popular. Accelerators contain cuda-and tensor-cores, which allow performing parallel operations on floating-point numbers and matrices. The performance of a graphics accelerator is defined as the sum of the productivity of all cores without taking into account performance losses on scheduling and interaction (4) [5].

P gpu = P cuda + P tensor , (4) where Pcuda is total performance of the graphics accelerator cuda-cores, Ptensortotal performance of tensor-cores of the graphics accelerator.

Let us determine the performance value of the cuda-cores of the graphics accelerator using formula (5), assuming that the floating-point operation is performed in one clock cycle.

Tensor-cores perform a multiplication of square matrices in one clock cycle. When calculating the number of operations performed in this case, we will take into account that the calculation of each element of the resulting matrix requires the execution of multiplication operations equal to the order of the matrix, as well as addition operations one less. Thus, the total performance of tensor-kernels is calculated as (6).

Note that the accuracy of performing floating point operations for different cores may differ. So, in the graphics accelerator NVidia Tesla V 100 cuda-cores work with double precision numbers, and tensor-cores with single precision numbers. In this method of performance evaluation, this feature is not taken into account.

The total performance of cuda-and tensor-cores are determined by formulas ( 5) and (6). P cuda = n cuda F gpu (5) P tensor = n tensor r 2 (2r − 1)F gpu (6) where ncuda is number of graphics accelerator cuda-cores, ntensornumber of tensor-cores of the graphics accelerator, rsquare matrix order, Fgpugraphics accelerator frequency. Thus, the peak performance of the graphics accelerator is calculated by the formula (7). 𝑃 𝑔𝑝𝑢 = (𝑛 𝑐𝑢𝑑𝑎 + 𝑛 𝑡𝑒𝑛𝑠𝑜𝑟 𝑟 2 (2𝑟 − 1))𝐹 𝑔𝑝𝑢 (7) The peak performance of the computing node of a hybrid high-performance computing cluster is calculated by the formula (8).

𝑃 ℎ𝑜𝑠𝑡 = 𝑁 𝑐𝑝𝑢 𝑛 𝑐𝑜𝑟𝑒 𝑛 𝑠𝑡𝑟𝑒𝑎𝑚 𝑛 𝑢𝑛𝑖𝑡 𝐹 𝑐𝑝𝑢 + (8) (𝑛 𝑐𝑢𝑑𝑎 + 𝑛 𝑡𝑒𝑛𝑠𝑜𝑟 𝑟 2 (2𝑟 − 1))𝐹 𝑔𝑝𝑢 The total peak performance of a hybrid high-performance computing cluster is calculated as (1). As shown above, the performance of the HPC cluster is calculated as the sum of the performances of its components and is expressed by the number of floating-point operations performed per second.

The resource of the hybrid high-performance computing cluster in the time interval will be the peak number of floating point operations available to users during this interval.

The total number of operations of the hybrid high-performance computing cluster Op(T) on the time interval T is defined as: 𝑂𝑝 (𝑇) = 𝑃 𝑝𝑒𝑎𝑘 𝑇, (9) where T is time interval.

The peak estimate differs from the actual, which is determined on the basis of various tests. However, as noted above, in this method we will use the peak values.

To estimate the requirements of applications to the resources of a hybrid high-performance computing cluster, we calculate the number of operations required for the execution of the application (10).

For each application, a number of CPU cores, graphics accelerators, and runtime are reserved. We take into account that the resources of graphic accelerators are reserved entirely, and the resources of central processors -by cores. Therefore, the total number of hybrid high-performance computing cluster operations performed by the task -Opapp(t) -for a given time t is determined by the number of cores of the central processors (Rcore) and graphics accelerators (Rgpu) reserved by the application.

𝑂𝑝 𝑎𝑝𝑝 (𝑡) = (

(𝑅 𝑐𝑜𝑟𝑒 𝑃 𝑐𝑝𝑢 ) 𝑛 + 𝑅 𝑔𝑝𝑢 𝑃 𝑔𝑝𝑢 )𝑡, (10)

where Rcore is the number of cores reserved by the application, Rgputhe number of graphics accelerators reserved by the application, ntotal number of cores in CPU.

After calculations for all applications i=1…N, the execution of which accounted for the period T, we obtain the total number of operations required for the execution of applications on the period T (11):

𝑂𝑝 𝑎𝑝𝑝 (𝑇) = ∑( (𝑅 𝑐𝑝𝑢 𝑃 𝑐𝑝𝑢 ) 𝑛 + 𝑅 𝑔𝑝𝑢 𝑖 𝑃 𝑔𝑝𝑢 )𝑡 𝑖 𝑁 𝑖=1 for 𝑡 𝑖 ∈ 𝑇(11)

Figure 1 shows a diagram of tasks performed on period T. Note that it is possible that only part of the execution time of the application falls on this period. In this case, when estimating the resources used, only the time interval ti belonging to T is taken into account. The Q(T) indicator can be used to set and evaluate the performance indicator of high-performance hybrid computing systems, plan the modernization of the cluster and draw up plans for the calculations of users of the hybrid cluster.

Note that the proposed assessment of the quality of loading of processor resources Q(T) is purely declarative and does not take into account the degree of use of allocated resources, the efficiency of algorithms and the quality of program code.

System of Dynamic Priorities

To provide an advantage to applications that use the resources of the cluster qualitatively, a system of dynamic priorities should be used. The system is based on the cluster utilization indicator.

To obtain an indicator of the use of computing resources, it is proposed to introduce the task profiling coefficient as an integral indicator of the quality of resource use (13).

𝑂𝑝 𝑎𝑝𝑝−𝑝𝑟𝑜𝑓 (𝑡) = 𝐾 𝑝𝑟𝑜𝑓 𝑂𝑝 𝑎𝑝𝑝 (𝑡) (13) where Kprof is profiling ratio.

Using such an assessment allows you to avoid a situation when the application does not use or irrationally uses the resources requested from the computing cluster. The coefficient of profiling is obtained by running a custom application under the control of a special debugging tool -a profiler that allows you to determine the degree of resource utilization, the execution time of individual code sections, bottlenecks and problems of memory usage. As part of development packages, there are profilers for both code executed on central processors and graphic accelerators.

Information on program profiling should be available both to the developer of a scientific application and to the division operating the computing cluster. This is necessary to take measures to improve the efficiency of the program code and increase the efficiency of the functioning of the hybrid HPC cluster as a whole.

Obviously, applications with a high profiling coefficient improve the quality indicator of a high-performance cluster workload. Therefore, a competitive advantage should be given to such applications. This encourages users to improve the calculation algorithms and taking into account the capabilities of the computing cluster. A classic way of encouraging tasks with a high profiling rate is to introduce a system of dynamic priorities based on the profiling coefficient.

The introduction of dynamic priorities allows within certain limits to change the priority of an application depending on its quality. This service policy is especially useful in conditions of heavy workload of the computing cluster. It allows to improve the quality of resource use and reduce the workload, as well as provide an advantage in the implementation of the applications that make the most use of the cluster's resources.

The decision to change the priority should be made on the basis of a comparison of the measured profiling coefficient with the recommended one, which is determined by expert. It is possible to set several threshold values of the profiling coefficient, for each of which there is a different priority rule. For example, for two quality thresholds (profiling coefficients К1, К2) that divide a multitude of applications into three subsets of quality "low", "medium", "high", the dynamic priority can be calculated based on a piecewise linear function (14).

𝑃𝑟 𝑑𝑦𝑛 = { 𝑃𝑟 𝑏𝑎𝑠𝑒 (С 0 𝐾 𝑝𝑟𝑜𝑓 − С 0 𝐾 1 + 1), 𝐾 𝑝𝑟𝑜𝑓 < 𝐾 0 𝑃𝑟 𝑏𝑎𝑠𝑒 (С 1 𝐾 𝑝𝑟𝑜𝑓 − С 1 𝐾 1 + 1), 𝐾 1 > 𝐾 𝑝𝑟𝑜𝑓 ≥ 𝐾 0 𝑃𝑟 𝑏𝑎𝑠𝑒 ( С 2 𝐾 𝑝𝑟𝑜𝑓 − С 2 𝐾 2 + С 1 𝐾 2 − С 1 𝐾 1 + 1), 𝐾 𝑝𝑟𝑜𝑓 ≥ 𝐾 1 (14)

where Prdyn is dynamic application priority;

Prbasebasic application priority; Kprofcoefficient derived from application execution profiling; K1, К2expert profiling coefficients; C0, C1, C2expert change factors. Figure 2 shows an example of the dependence of dynamic priority on the values of К and С with Prbase =1. Thus, when obtaining the values of the profiling coefficient below K1, a linear decrease in priority relative to the base value is made; when K1 is exceeded, a linear increase in priority is obtained. If K2 is exceeded, the priority growth increases. The recommended profiling coefficients K and coefficients C are determined by an expert method, based on the characteristics of the functioning and loading of the computing cluster. The choice of the recommended parameters K and C should be made by the owner of the HPC cluster and proceed from the following.

In the early stages of cluster operation, when technologies and algorithms are being debugged, the priority should be changed to the minimum extent. The requirements for grading factors should not be too high. Therefore, the values of C, which determine the slope of the straight lines, should be chosen closer to zero, this provides a slight change in priorities.

With increasing workload on the computing cluster and the need for more accurate task management, the values of C can be increased. This leads to a more significant change in priority compared to the baseline with a significant deviation of Kprof from K1.

Conclusion

The proposed methodology for estimating the workload on the hybrid HPC cluster allows to determine how fully and efficiently the resources of the hybrid cluster are used. On the basis of the results obtained, it is possible to determine indicators of ROI, plan the work of the cluster, and determine the need for modernization.

The system of dynamic priorities will allow to control the quality of resource utilization of hybrid highperformance computing clusters when they perform different types of applications from various fields of science and technology.