THE CMS TIER1 AT JINR: FIVE YEARS OF OPERATIONS

Keywords

• WLCG
• Tier1
• grid
• CMS
• distributed computing

1. Introduction

The LHC accelerator at CERN [1] started to work in November 2009. Since then hundreds of Petabytes (~50-70 PB/year) of data – raw, processed, and simulated – were gathered from all experiments. To process and analyze such an unprecedented volume of data, the largest in the world computing infrastructure was built which comprises of more than 170 computer centers in 42 countries – Worldwide LHC Computing Grid network (WLCG [2]). The WLCG is set up as a layered structure: the 14 of all the computer centers, which make the layer 1 (Tier1), receive raw data from layer 0 (Tier0) at CERN in real time to store and preprocess it and provide the centers of layer 2 (Tier2) with it for analysis. The Joint Institute for Nuclear Research (JINR [3]) is one of Tier1 sites for CMS experiment [4] at LHC [5], and one of Tier2 sites for all four experiments at LHC and other Virtual Organizations (VO) of the worldwide GRID infrastructure.

The Tier1 at JINR started as a prototype node in 2011-2013 as the Federal Target Programme Project: «Creation of the automated system of data processing for experiments at the LHC of Tier1 level and maintenance of Grid services for a distributed analysis of these data». The project assumed the deployment of distributed Tier1 center in Russia. The primary producers of experimental data at LHC are two largest experiments – CMS and ATLAS [8]. This was the reason which defined the predestination of the Russian Tier1. Two Tier1 sites were established: at the Laboratory of Information Technologies (LIT) of JINR to deal with the CMS data, and at NRC “Kurchatov Institute” [9] to support ATLAS, ALICE [10] and LHCb [ 11 ] experiments. The primary reason of assigning the CMS support to JINR is the fact that Russian institutions participating in CMS and JINR as international organization with its member states are represented in the experiment as united collaboration - “Russia and Dubna Member States” (RDMS) [ 12 ]. More than 300 scientists and specialists from 21 institutions of 8 countries participate in RDMS. This way of cooperation of many institutions of various specializations allows RDMS to be completely responsible for a number of detectors in CMS, and also to make significant contribution to the development of the basic software and computing.

The basic tasks of the Tier1 center for CMS at JINR (T1_RU_JINR in WLCG mnemonics) are: 1) to receive the experimental data from Tier-0 site in the volume determined by the WLCG agreement (WLCG MOU); 2) archiving and custodial storage of part of experimental RAW data; 3) consecutive and continuous data processing; 4) additional processing (skimming) of RAW, RECO (RECOnstructed) and AOD (Analysis

Object Data) data; 5) data reprocessing with the use of new software or new calibration and alignment constants of parts of the CMS setup; 6) making available AOD data-sets; 7) serving RECO and AOD datasets to other Tier1/Tier2/Tier3 sites for their duplicated storage (replication) and physical analysis; 8) running production reprocessing with the use of new software and new calibration and alignment constants of parts of the CMS setup, protected storage of the simulated events; 9) production of simulated data and data analysis recorded by the CMS experiment.

2. Tier1 infrastructure

Tier-1 deployment required substantial upgrade of JINR LIT data center infrastructure to grant not only enough computer power with modern software to store and process the data but also evolved and durable support facilities to provide fail-safe 24/7/365 functionality of the site.

2.1. Engineering infrastructure

Since 2014 intensive works were carried out at LIT on the engineering infrastructure upgrading. New closed-loop air conditioning system was set up. It is comprised of two UNIFLAIR ARAF 1204А chillers working in N+1 redundancy. Eight APC InRow ACRC 502 precise air conditioners of 39.56 kW cold productivity provide the required air temperature and humidity in closed aisle of Tier1 setup. Two 300 кWA Galaxy 7000 UPS guarantee the fail-safe power for the setup. In 2018 two diesel generators were put into operation to guarantee the autonomy of Tier1 power supply.

2.2. Networking

One of the most important components of JINR Tier1 providing access to resources and the possibility to work with the Big Data is the network infrastructure. External optical telecommunication channel uses DWDM (Dense Wave Division Multiplexing) technology for data transmission. Ethernet frames are transmitted through DWDM carrier signals. The most attractive feature of DWDM technology in comparison with traditional optical technologies, in which one optical fiber is transmitted only one digital signal, is the parallel transmission of multiple digital signals over the same optical fiber. Optical equipment from Nortel Systems (Canada) was installed at three points of the JINR-Moscow data channel: at the LIT JINR central telecommunications center, at the intermediate point of the optical highway (Radishchevo settlement), at the Moscow International Exchange (Moscow Internet Exchange) Site Internet. This equipment operates with 2λ equal to 10 Gbit/s each and 1λ to 100 Gbit/s. Thus, the capacity of the optical telecommunication channel of JINR is 120 Gbit/s.

The external superimposed network LHCOPN [ 14 ] (JINR-CERN) passing through MGTS-9 in Moscow, Budapest, Amsterdam, is for communication of the Tier0 (CERN) and Tier1 (JINR) centers, while the external superimposed network LHCONE [ 15 ], passing the same route, is intended for communication of the Tier2.

The Tier1 network segment provides fail-safe operation of 160 disk servers, 25 computer blade servers, 100 Grid infrastructure support servers, and tape robot. The segment is built with Brocade equipment in which the IS–IS (Intermediate System to Intermediate System) protocol is used for network segment definition on level 2 of OSI (Open Systems Interconnection) model.

For this protocol Dijkstra's algorithm from graph theory is used. It compares and calculates the shortest path through all nodes in the network. It is constructing a shortest-path tree from the first vertex to every other vertex in the graph. On this basis was developed a modern protocol Transparent Interconnection of Lots of Links (TRILL) on the second tier of the OSI model, enabling building solutions for campus networks and data centers. Transparent Interconnection of Lots of Links (TRILL) offers many advantages. TRILL provides Layer 2 multipath and multi-hop routing.

A full redundancy of links is provided for at all levels. As a result of such architecture a failure of one switch shall lead to the reduction in the total traffic capacity of the network segment only by 25%. In such case all servers will have access to the external network [ 16 ].

2.3. Hardware

The evolution of computing and storage capacity of CMS Tier1 at JINR [ 17-19 ] from 2014 to 2018 is presented on Figure 1.

Currently, the JINR Tier1 for CMS hosts 275 worker nodes (WNs) for a total amount of 4720 computing slots and a power capacity of 72,3k HS06. All the computing resources are centrally managed by a single batch system, Torque 4.2.10 (home made) and Maui 3.3.2 (home made). For batch processing support a server with the cluster resource manager system and task planner is set up. It should be noted that our Tier1 is configured to handle only 10-core pilots.

As a Tier1 we are responsible not only for data processing but also data storage. We should support from raw data (including MC data) storage to AOD production and storage, and storage of physics data used in end user analysis. Based on those tasks the storage system must meet the following requirements: • scalability up to hundreds PB, • high availability (for data taking and processing), • data-intensive processing with high I/O performance, • data portability to Grid service.

So, the storage system is comprised of two parts: disk storage for data processing, and tape storage for long-time data keeping. JINR Tier1 operates a large storage infrastructure based typically on Supermicro and DELL disks servers and tape storage IBM TS3500. Currently there are ~ 7 PB of disks space (SE Disk only), ~1 PB disks buffer space for ~10 PB tape storage (SE Buffer + Tape).

The data can be accessed through standard protocols and interfaces, as defined by the WLCG projects (GridFTP and SRM, XRootD). dCache-3.2 software are used for disk storage system and Enstore 4.2.2 for tape robot. To support storage and data access systems, 8 physical and 14 virtual machines are installed.

Concerning Tier1 data exchange we can say that 157 sites worldwide read data from our SE Buffer+Tape, 140 sites write new file. Leaders are CERN, KIT(Germany), RAL (UK). 316 sites worldwide read data from our SE Disk only and 150 sites write new files. In 2018 data exchange to SE disk only was 36.50 PB (11.19 PB new files), to SE Buffer + Tape was 6.78 PB (3.38 PB new files, which is two times as much as in 2017. Figure 2 shows the monthly volumes of data exchange with the outer world.

The CMS PhEDEx (Physics Experiment Data Export) [ 20 ] is responsible for large-scale data transfers across the grid ensuring transfer reliability, enforcing data placement policy, and accurately reporting results and performance statistics. The system has evolved considerably since its creation in 2004, and has been used daily by CMS since then CMS PhEDEx is used for datasets transfer. Figure 3 shows the worldwide dataset transfers from T1_RU_JINR and to T1_RU_JINR since the start of operation. Information is collected from PhEDEx – CMS Data Transfers dashboard [ 21 ].

We have experienced several difficulties while using hardware RAID6 on FST pools. It provides only one-way data consistency check (VD health status and PD presence). There are no S.M.A.R.T data collection available, no info about physical drive (PD) temperature/health. Pre-fail condition HDD’s may suddenly faults while rebuilding is already in progress and more or equal 3 faulted HDD’s in one pool makes data unavailable.

With transition to ZFS RAIDZ2 we have two-way data consistency check. In HBA mode S.M.A.R.T of each installed HDD is available, and we can collect several parameters: reallocated sectors count, pending sectors count, HDD temperature, read uncorrected errors count, write uncorrected errors count, ECC correction algorithm invocations count, and overall health status. ZFS provides information about pool and devices: “State” to indicate the current health of the pool, “Status” to describe what is wrong with the pool, “Scrub” to identify the current status of a scrub operation, and “Errors” to identify known data errors or the absence of them. Thus, we can use both of that information to find faulting PDs and replace them in advance.

Rebuild process (resilvering) for 1 or 2 HDD’s in ZFS is almost the same as for RAID6. In our configuration we scheduled every week pool-scrub to check data consistency. Resilvering operation also takes about 1 week for 6TB/8TB HDD’s. If ZFS automatically marks another one HDD as fault, while resilvering is incomplete, it looks like faulted HDD is usually still present in pool, but in read only state. In such a case we can successfully finish previous operation before starting another one, and finally lose only several files instead of tens TB of data.

For Tier1 operation, WLCG grid environment support services are required. Some WLCG services have been installed on physical machines, the other – on virtual ones. The WLCG services are installed with software EMI-3 for compatibility with software grid-environment of WLCG. Currently, 21 services are installed. The services provide the entire infrastructure of remote work with the grid, namely:  user and virtual organizations (VO) authorization,  task run from VO remote services,  the WLCG information system,  different algorithms of remote testing and verification of the service environment on local resources.

3. Tier1 hardware and services monitoring

For a robust performance of the complex it is necessary to monitor the state of all nodes and services – from the supply system to the robotized tape library. We developed special monitoring system for our Tier1 [ 22 ]. The monitored data are collected from the wide range of hardware and software related to Tier1: cooling systems, temperature sensors, uninterruptable power supplies (UPS), computing servers, disk servers, managing services, L2 and L3 switches/routers, and tape robot. More than 850 nodes are under observation, ~8000 checks were performed in real time, and ~100 scripts were written to support this system based on Icinga2 [ 23 ]. The created monitoring system allows one not only to control the state of computing nodes and disk servers, but also to obtain information about the number of executed and waiting in the queue jobs, the Tier1 CPU load (Figure 4), local network load and incoming and outgoing data streams (Figure 5).

Apart from hardware metrics, service metrics are scattered among many internal and external systems. This information relates to data transfers, data storage, and data processing. In order to keep track of the services administrator should regularly check several dozens of web pages on common experiment and WLCG dashboards. Interpretation of data is more complex. To provide a single source of aggregated monitoring information and to perform basic analysis of data, and provide status of the system the special service monitoring system was developed [ 24 ]. Now we can collect data from four types of sources: JSON data, HTML data, Databases and command line. This collected data information is relevant only for our Tier1, stored in the database and shown on the web-page.

Other little, but helpful issue is connected to PhEDEx system, which was designed to operate mostly automatically. But sometimes, due to different reasons it requires intervention to fix errors manually. Source of information about errors is a corresponding PhEDEx webpage. Every error is a big form with source/destination site, time of assigned/start/done, PFNs to/from, transfer/detail/validate logs. In order to simplify operation python script was written to list important errors and provide relevant information about them. This script allows one to detect next types or errors: nsf - no such file or directory, csmm – checksum mismatch, smm – size mismatch, and uto – user timeout over. Finally the list of files in error state is produced.

4. Tier1 performance results

Since commissioning, the JINR Tier1 has steadily increased its productivity and maintained the level of availability and reliability required by the CMS experiment. The Figure 6 shows JINR Tier1 (blue) monthly reliability results compared to the average Tier1 reliabilities (orange) as well as with the WLCG target for site reliability (yellow area), which is set to 97% since 2009, according to the WLCG MoU.

The contribution of the Tier1 center in JINR to the processing of CMS experiment data for the last 5 years is shown in Figure 7 as a percentage of the total number of jobs and processed events by all Tier1 sites. The percentage of jobs has been increasing during the year by year and reached about 13% of total CMS jobs and about 20% of good events processed. In total, since the beginning of operation, 27 722 950 jobs have been completed and 308 560 399 956 events processed on our site.

Organized data processing activities are carried on at the Tier1 centres by the Data Operations team. These include reprocessing of raw data, production of AOD, analyzing, large scale physics skims and processing. During this years our Tier1 processed 281 555 814 018 events by analysis jobs, 10 042 089 961 events were reprocessed, 5 569 434 071 events were processed through production activity, etc. Figure 8 shows the distribution of completed jobs by activities.

Figure 9 shows the number of CMS events processed in Million events at our Tier1 in August 2018 and figure 10 shows the number of jobs processed in the same period of time [25].

5. Conclusion

During the last five years the JINR Tier1 centre for CMS experiment at CERN progressively increased its scale and improved its reliability. It creates conditions for physicists from JINR and its JINR Member States, RDMS-CMS collaboration for a full-scale participation in processing and analysis of data of the CMS experiment on the Large Hadron Collider.

The invaluable experience of launching the Tier1 center will be used for creating a system of storage and data processing of megaproject NICA and other scale projects of the JINR-participating countries. [1] CERN European Organization for Nuclear Research, URL https://home.cern [2] WLCG Worldwide LHC Computing Grid, URL http://wlcg.web.cern.ch/ [3] JINR Joint Institute for Nuclear Research, URL http://www.jinr.ru/ [4] CMS Compact Muon Solenoid experiment, URL https://cms.cern/detector [5] LHC – The Large Hadron Collider, URL http://lhc.web.cern.ch [8] ATLAS Experiment, URL http://atlas.cern/ [9] National Research Center “Kurchatov Institute”, URL http://www.nrcki.ru/ [10] ALICE A Large Ion Collider Experiment, URL http://aliceinfo.cern.ch/Public/Welcome.html

References

[11] LHCb Large Hadron Collider beauty experiment , URL http://lhcb-public.web.cern.ch/lhcbpublic/

[12] RDMS: Russia and Dubna Member States CMS Collaboration , URL http://rdms-cms.jinr.ru/

[13] CMS: The Computing Project Technical Design Report , URL http://cdsweb.cern.ch/ record/838359/ files/lhcc-2005-023.pdf

[14] The Large Hadron Collider Optical Private Network URL http://lhcopn.web.cern.ch/lhcopn/

[15] The Large Hadron Collider Open Network Environment URL http://lhcone.web.cern.ch/

[16] S. Baginyan , A. Dolbilov , V. Korenkov , Network for datacenter Tier 1 at JINR for experiment CMS (LHC ), T-Comm Vol. 10 , No. 1 , 25 - 19 ( 2016 )

[17] N.S. Astakhov , S.D. Belov , P.V. Dmitrienko , A.G. Dolbilov , I.N. Gorbunov , V.V. Korenkov , V.V. Mitsyn , S.V. Shmatov , T.A. Strizh , E.A. Tikhonenko , V.V. Trofimov , V.E. Zhiltsov , CMS Tier1 at JINR , Proc. NEC' 2013 , p. 5 ( 2014 )

[18] N.S. Astakhov , A.S. Baginyan , S.D. Belov , A.G. Dolbilov , A.O. Golunov , I.N. Gorbunov , N.I. Gromova , I.S. Kadochnikov , I.A. Kashunin , V.V. Korenkov , V.V. Mitsyn , I.S. Pelevanyuk , S.V. Shmatov , T.A. Strizh , E.A. Tikhonenko , V.V. Trofimov , N.N. Voitishin , V.E. Zhiltsov , JINR Tier- 1 centre for the CMS experiment at LHC, Particles and Nuclei, Letters, v. 13 , 5 ( 203 ). 1103 - 1107 ( 2016 )

[19] N.S. Astakhov , A.S. Baginyan , S.D. Belov , A.G. Dolbilov , A.O. Golunov , I.N. Gorbunov , N.I. romova, I.S. Kadochnikov , I.A. Kashunin , V.V. Korenkov , V.V. Mitsyn , I.S. Pelevanyuk , S.V. Shmatov , T.A. Strizh , E.A. Tikhonenko , V.V. Trofimov , N.N. Voitishin , V.E. Zhiltsov , Tier-1 CMS at JINR : Status and Perspective . CEUR Workshop Proceedings V1787, P. 1 - 14 ( 2016 )

[20] R. Egeland , T. Wildish and S. Metson . Data transfer infrastructure for CMS data taking, PoS(ACAT08)033 pdf ( 2008 )

[21] PhEDEx - CMS Data Transfers , URL https://cmsweb.cern.ch/phedex/prod/Info::Main

[22] I.A. Kashunin , A.G. Dolbilov , A.O. Golunov , V.V. Korenkov , V.V. Mitsyn , T.А. Strizh, Proc. of the 7th International Conference Distributed Computing and Grid-technologies in Science and Education, CEUR-ws . org/ Vol- 1787 / 256 -263-paper-43.pdf, ( 2016 )

[23] Icinga , URL https://icinga.com/

[24] I.S. Kadochnikov , I.S. Pelevanyuk JINR Tier1 service monitoring system: Ideas and Design , Proc. of the 7th International Conference Distributed Computing and Grid-technologies in Science and Education , http://ceur-ws. org/ Vol- 1787 / 275 -278-paper-46.pdf, ( 2016 )