JINR WLCG TIER1 & TIER2/CICC ACCOUNTING SYSTEM

Abstract

The problem of evaluating the efficiency of the JINR MLIT grid sites has always been topical. At the beginning of 2021, a new accounting system was created, it managed to fully cover the functionality of the previous system and further expand it. The article provides detailed information on the implemented accounting system.

Keywords

• Accounting
• WLCG
• Grid
• Monitoring

1. Original accounting system

Since the beginning of 2001, MLIT JINR started creating a distributed system for processing, storing and analyzing experimental data from the experiments at the Large Hadron Collider (LHC) using grid technologies [ 1 ]. In 2003, the Russian segment of the LCG Global Infrastructure was organized under the Russian Data Intensive Grid Consortium (RDIG) [ 2 ], and the Tier2 grid site for data processing within the distributed computing infrastructure began functioning at JINR. The Linux operating system, the Torque batch processing system [ 3 ], the Maui task scheduler [ 4 ] and the software stack that ensures consistent work within the distributed grid infrastructure are the main software of the computing cluster. The Torque and Maui systems were finalized at MLIT and, by default, had a built-in script for collecting statistics on the batch system. The script was run with various parameters at a certain time and generated text files (Fig. 1) containing information about the operation of the system. The collected data was used to evaluate the effectiveness of jobs performed on the grid site, to account for them by users and compile various reports. Jobs were grouped according to their belonging to experiments and virtual organizations included in the RDIG, data processing from which was carried out on the site (lalice – Alice experiment at the LHC, lcms – CMS experiment at the LHC, etc.).

2. Prerequisites for the development of a new system

In mid-2020, due to the obsolescence and lack of support for Torque and Maui, it was decided to switch to a new cluster management and task dispatching system for large and small Linux clusters, SLURM [ 5 ]. Unlike Torque and Maui, SLURM does not allow a number of different default parameters to be displayed: ● CPUclock – CPU time spent on job execution for a certain period of time; ● Wallclock – total astronomical time spent by the job for a certain period of time; ● efficiency of using the cluster computing resources by the job.

These parameters can be calculated from the database (DB) of the SLURM system, which is formed during the operation of the system. To do it, there is a need to write a special script and create your own accounting system, which will allow obtaining the necessary parameters upon request.

3. Development of an accounting system

At the end of 2020, the SLURM system was put into operation. At the same time, the development of an accounting system started. The initial task was to develop a console version that would display data for a certain period of time in a tabular format on the screen. Similar to the original accounting system, this data can be stored in the form of reports. The main parameters that the accounting system should display for a certain period of time are:  number of jobs grouped by the affiliation to an experiment/group;  CPUclock of grouped jobs;  Wallclock of grouped jobs;  average number of cores used by the job in a group;  efficiency of using the cluster computing resources by a job group.

Terms such as CPUclock and Wallclock have analogs, i.e. total_cpu_time and elapsed_time in SLURM, respectively. SLURM has a special command sacct to display various system parameters on the screen. The output is in a tabular format, where the required parameters are displayed for each job. Due to this, it is possible to calculate the amount of CPUclock for a job group as the sum of total_cpu_time for each job for a certain period of time. Wallclock is calculated in the same way, only in this case the amount is calculated from elapsed_time. The efficiency of using the cluster computing resources by a job group can be calculated as the percentage of these parameters to the number of cores allocated for each job. This parameter is especially useful for detecting cases when a job reserves cores without using all the processor power. Thus, the poorly optimized code can be identified. Batch queues can also comprise jobs with a different number of cores required for computing. The script of the accounting system was developed taking into account this feature.

As a result, an accounting system script was created, it allows displaying the main parameters on the screen (Fig. 2).

The next task is to develop a visualization system. Currently, there are various options for choosing software for data visualization systems. Since 2014, a monitoring system [ 6 ] has been operating at the Multifunctional Information and Computing Complex (MIСС) [ 7 ], and one of its components is the Grafana visualization system [ 8 ]. It was decided to build a visualization system on its basis. That enabled easier integration into the existing monitoring system.

To display statistics, the Grafana system receives data from a specific resource, i.e. a backend (database, software, socket, etc.). One of them is the MariaDB database [ 9 ]. To connect the accounting system with Grafana, special support for writing data to the accounting system script was added. The script is launched by the operating system through the standard service cron [ 10 ] and writes the parameters for the day, week, month and year to the database.

4. Vizualization system

An interactive visualization system was created on the basis of Grafana, it allows providing the user with the most up-to-date analytical data:  graphs and pie charts of CPUclock and Wallclock for a certain period of time;  graphs for the use of the cluster computing resources by jobs;  graph of the number of completed jobs;  tabular version of the displayed data.

As a result, an information display (Fig. 3) was created. All calculations of the accounting system are carried out taking into account only completed jobs. The longer the time interval for completed jobs accounting is, the more accurate the report data is. ● Сlock_type

It displays the change in the type of parameters, such as Wallclock, CPUclock and their versions translated to HepSpec06 [ 11 ]. After changing this parameter, the top-most graph and pie charts change according to the selection. ● Data – time interval

The date of both the beginning and the end of the data display can be set. Changing the date affects the display of all graphs and tables.

Thus, by changing various parameters, the set of data required for displaying can be configured.

5. Conclusion

In February 2021, various tests and the verification of the functionality of the accounting system, including data verification with the EGI Federation account [ 12 ], were performed. The test results showed that the difference between the calculations in the systems was 0.4%, which can be caused by different time frames and calculation algorithms.

In March 2021, the accounting system was integrated into the general monitoring system of the MICC.

Access to the pages of the accounting system for the WLCG (Worldwide LHC Computing Grid) sites [ 13 ], Tier1, Tier2 of MLIT JINR, is carried out via the main page of the Litmon monitoring system (Fig. 4). As a result, it became possible to back up the readings of the other accounting systems, as well as to display JINR specific tasks that are performed on the sites and related to tasks within the NICA project. The Tier1, Tier2 accounting system was put into operation. It completely covered the functionality of the original system and significantly expanded it due to the flexible configuration of the visualization system. The data collected by the original accounting system since 2018 was imported into the visualization system. at:

References

[1] V. Korenkov , E. Tikhonenko, GRID Concept and Computer Technologies in the LHC Era , Physics of Elementary Particles and Atomic Nuclei (PEPAN) , vol. 32 , p. 6 , 2001 , in Russian.

[2] A. Soldatov , V. Korenkov , V. Ilyin , Russian Segment of the LCG Global Infrastructure , Open Systems, N1 , 2003 , in Russian.

[3] Torque . Available (accessed 15.07 . 2021 ).

[4] Maui . Available (accessed 15.07 . 2021 ). at: http://dipc.ehu.es/cc/computing_resources/jobs/batch_systems/torque/ http://dipc.ehu.es/cc/computing_resources/jobs/batch_systems/torque/

[5] SLURM. Available at: https://sLURM.schedmd.com/documentation. html (accessed 15.07 . 2021 ).

[6] I. Kashunin , V. Mitsyn , V. Trofimov , A. Dolbilov . Integration of the Cluster Monitoring System Based on Icinga2 at JINR LIT MICC / / PEPAN Letters: http://www1.jinr.ru/Pepan_letters/panl_2020_3/14_kashunin.pdf, 2020 .

[7] Dolbilov А. , et al. Multifunctional Information and Computing Complex of JINR: Status and Perspectives // CEUR Workshop Proc . 2019 . V. 2507. P. 16 ‒ 22 .

[8] Grafana . Available at: https://grafana.com / (accessed 15.07 . 2021 ).

[9] MariaDb. Available at: https://mariadb.org / (accessed 15.07 . 2021 ).

[10] Crontab . Available at: https://ru.wikipedia.org/wiki/Cron (accessed 15 .07. 2021 ).

[11] HepSpec06 . Available at: https://www.gridpp.ac.uk/wiki/HEPSPEC06 (accessed 15 .07. 2021 ).

[12] EGI . Available at: https://accounting.egi. eu/ (accessed 15.07 . 2021 ).

[13] Worldwide LHC Computing Grid . Available at: https://wlcg.web. cern.ch/ (accessed 15.07 . 2021 ).