=Paper=
{{Paper
|id=Vol-2023/23-29-paper-4
|storemode=property
|title=High performance computing system in the framework of the Higgs boson studies at ATLAS
|pdfUrl=https://ceur-ws.org/Vol-2023/23-29-paper-4.pdf
|volume=Vol-2023
|authors=Nikita Belyaev,Alexei Klimentov,Rostislav Konoplich,Dimitrii Krasnopevtsev,Kirill Prokofiev,Vasily Velikhov
}}
==High performance computing system in the framework of the Higgs boson studies at ATLAS==
https://ceur-ws.org/Vol-2023/23-29-paper-4.pdf
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017
HIGH PERFORMANCE COMPUTING SYSTEM IN THE
FRAMEWORK OF THE HIGGS BOSON STUDIES AT ATLAS
N. L. Belyaev1,2,a, A. A. Klimentov1,3, R. V. Konoplich4, 5,
D. V. Krasnopevtsev1,2, K. A. Prokofiev6, V. E. Velikhov1, on behalf of the
ATLAS Collaboration
1
National Research Center “Kurchatov Institute”, 1 Akademika Kurchatova pl., Moscow, 123182,
Russia
2
National Research Nuclear University MEPhI (Moscow Engineering Physics Institute), 31
Kashirskoe highway, Moscow, 115409, Russia
3
Brookhaven National Laboratory BNL, Upton, Suffolk County, New York, USA
4
New York University, 4 Washington Place, New York, NY 10003, USA
5
Manhattan College, 4513 Manhattan College Parkway, Riverdale, New York, NY 10471, USA
6
Department of Physics and Institute for Advanced Study, the Hong Kong University of Science and
Technology, Clear Water Bay, Kowloon, Hong Kong, China
E-mail: a Nikita.Belyaev@cern.ch
Higgs boson physics is one of the most important and promising fields of study in modern high
energy physics. To perform precision measurements of the Higgs boson properties, the use of fast
and efficient instruments of Monte Carlo event simulation is required. Due to the increasing amount
of data and to the growing complexity of the simulation software tools, the computing resources
currently available for Monte Carlo simulation on the Large Hadron Collider (LHC) Grid are not
sufficient. One of the possibilities to address this shortfall of computing resources is the usage of
institutes’ computer clusters, commercial computing resources and supercomputers. In this paper, a
brief description of the Higgs boson physics, Monte Carlo generation and event simulation
techniques are presented. A description of modern high performance computing systems and tests of
their performance are also discussed. These studies have been performed on the Worldwide LHC
Computing Grid and Kurchatov Institute Data Processing Center, including Tier-1 WLCG sites and
the OLCF Titan supercomputer. Monte Carlo simulated events produced with the Titan
supercomputer were used in the Higgs boson analysis and the results have been published by the
ATLAS collaboration.
Keywords: high performance computing, ATLAS, Higgs, CERN, supercomputers, data,
LHC, physics, reconstruction, GRID, cloud computing, WLCG, Tier-1, Titan
© 2017 Nikita L. Belyaev, Alexei A. Klimentov, Rostislav V. Konoplich, Dimitrii V. Krasnopevtsev,
Kirill A. Prokofiev, Vasily E. Velikhov
23
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017
1. High performance computing in high energy physics
The observation of a new particle compatible with the Standard Model (SM) Higgs boson by
the ATLAS and the CMS experiments [1, 2] has been an important step towards the validation of our
understanding of nature. This observation is based on the information from the Large Hadron
Collider (LHC). Experimental data was obtained during the first LHC data taking period (Run 1) in
2011-2012. Other impressive discoveries based on the LHC data include the pentaquark observation
[3] and the observation of CP violation in rare baryon decays [4], both by the LHCb experiment.
To perform more detailed studies in these areas and to probe the new physics beyond the
Standard Model (BSM), theoretical predictions for the physics processes under consideration must be
obtained. The most convenient way to obtain such predictions is to use Monte Carlo (MC)-generated
events. MC simulations are based on concrete theoretical models, which describe the processes under
consideration.
MC simulations usually require a large amount of computing resources. Generally, there are
two reasons for that. First, modern particle physics theories become more and more complex: they
are based on non-trivial mathematics and calculations required by these theories are resource-
intensive. Second, as far as measurements in particle physics are performed by the particle detectors,
the detector effects must be also taken into account. The ATLAS detector [5] consists of millions of
structural elements (more than 150M sensors). The response of each of them must be modelled and
taken into account. For the physics analyses, it is necessary to perform so-called Full Simulation of
the detector, which takes into account all the detector effects [6].
The resource intensity of MC tasks can be illustrated in terms of statistics. Figure 1 shows the
fraction of CPU time consumption of all jobs submitted to Worldwide LHC Computing Grid
(WLCG) by all its users during the first 35 weeks of 2017. The jobs were categorized according to
their type: Data processing, Validation, Testing, MC Simulation, MC Simulation Full, MC
Simulation Fast, MC Event Generation, MC Reconstruction, Group Production and Others. This plot
shows that MC-related jobs take more than 85% of all CPU time provided by the WLCG.
Figure 1. Consumption of CPU time for all WLCG jobs in seconds, obtained with the ATLAS
Dashboard [7]
24
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017
2. Higgs boson physics
Higgs boson physics is one of the most promising fields of study in modern particle physics.
However, this topic is not only promising, but also complex. The most intensive channel of the Higgs
boson production at the LHC in high energy proton-proton collisions is the gluon-fusion production
(ggF). The Feynman diagram of such a process at the leading order (LO, first-order terms of
Lagrangian in the Perturbation theory of Quantum Chromodynamics [8]) is shown in Figure 2.
Figure 2. Feynman diagram of the LO ggF Higgs boson production. Two gluons, emitted by protons,
interact and then produce the Higgs boson through the loop of t- or b-quarks [9]
When the center-of-mass energy of two protons is about 10 teraelectronvolts (TeV) or higher,
more than 90% of all Higgs bosons are produced by the ggF mechanism. The process shown in
Figure 2 is relatively simple, because there are no outcoming particles except the Higgs boson and
there are no hadron jets (so-called 0-jet final state).
Considering second-order terms of the Lagrangian (Next-to-leading order corrections, NLO),
the situation becomes more complex. Figure 3 presents some of the Feynman diagrams of ggF Higgs
boson productions in the NLO.
Figure 3. Feynman diagrams of the NLO ggF Higgs boson production. Higgs boson is still being
produced from t- or b-quark loops, but now the final state might contain one or two additional hadron
jets [9]
The complexity of NLO calculations is directly correlated with the number of jets. In NLO
case, 0-jet category consists of 29 Feynman diagrams. The generation time of 100k events at CERN
on LXPLUS machines [10] is about 10 minutes. The 1-jet category consists of 1050 diagrams and the
calculation time increases up to about 2 hours. The 2-jet category consists of 21510 diagrams and the
calculation time increases even more, up to about 24 hours. For the ggF Higgs boson production
NLO corrections can contribute up to 45% to the cross sections and thus must be taken into account.
25
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017
3. Role of High Performance Computers’ in HEP
High Performance Computers (HPC) are presently the most valuable instruments for CPU-
intensive tasks, such as Higgs boson production simulation with NLO corrections. One of the most
impressive realizations of supercomputer-based HPC system for science tasks, including high energy
physics, is the OLCF Titan, launched in 2012 by Oak Ridge National Laboratory, the performance of
which is higher than the computing power of all the ATLAS resources in WLCG [10]. Some
technical characteristics of the Titan supercomputer are listed below [11]:
• 27 PFLOPS (Peak theoretical performance);
• Cray XK-7 18,688 compute nodes with GPUs;
• 299,008 CPU cores AMD Opteron 6274 @2.2 GHz (16 cores per node);
• 32 GB RAM per node;
• NVidia TESLA K20x GPU per node;
• 32 PB disk storage (center-wide Luster file system);
• More than 1TB/s aggregate FS throughput;
• 29 PB HPSS tape archive;
The Titan supercomputer was used to perform the simulation of the Higgs boson production
associated with two hadron jets at NLO level with consequent decay of the Higgs boson to four
leptons: 𝑝𝑝 → 𝐻𝑗𝑗 → 4ℓ𝑗𝑗. Some technical parameters of this simulation are listed in Table 1,
namely: nFiles is the number of files in the input dataset; nEventsPerInputFile is the number of
events per input file and nEventsPerJob is the number of events per job. Expected CPU time is
estimated by internal GRID benchmarks based on previous information about the occupancy of the
particular GRID resource [12].
It is important to note, that while with CERN LXPLUS machines the generation event rate is
about 2.5k events/hour, with the Titan it is about 650k/hour.
Table 1. Parameters of MC Full Simulation of the Higgs boson production with Titan
CPU Time, HS06 seconds [13]
nEventsPerInputFile nEventsPerJob nFiles
Expected Total
2000 100 3000 72048000000 2284598809
The datasets produced with the Titan supercomputer were subsequently used by the ATLAS
collaboration for physics analysis, with the results of this study presented in the public note [14]. The
main purpose was to probe the separation power of BDT discriminants to distinguish ggf and vector
boson fusion (VBF) signals. As an example of obtained results, Figure 4 shows the distribution of the
Boosted Decision Tree (BDT) discriminant for a possible detector layout for the ATLAS HL-LHC
upgrade. The separation between the red and blue distributions visualises the ggf-VBF separation.
Figure 4. BDT classifier distributions for one of possible detector layouts [14]
26
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017
4. Cloud computing at Tier-1 clusters
The resource-intensive tasks can be handled not only by supercomputers, but also by WLCG
computing facilities and cloud computing. WLCG is made up of four layers, or "tiers"; 0, 1, 2 and 3.
Each tier provides a specific set of services:
● Tier 0 corresponds to the CERN Data Centre, which is located in Geneva, Switzerland and also
at the Wigner Research Centre for Physics in Budapest, Hungary. The two sites are connected
by two dedicated 100 Gbit/s data links. All data from the LHC passes through the central
CERN hub, but CERN provides less than 20% of the total computer capacity. Tier 0 is
responsible for the safe-keeping of the raw data (first copy), first pass reconstruction,
distribution of raw data and reconstruction output to the Tier 1s, and reprocessing of data
during LHC down-times.
● Tier 1 corresponds to thirteen large computer centres with sufficient storage capacity and with
round-the-clock support for the Grid. They are responsible for the safe-keeping of a
proportional share of raw and reconstructed data, large-scale reprocessing and safe-keeping of
corresponding output, distribution of data to Tier 2s and safe-keeping of a share of simulated
data produced at these Tier 2s.
● Tier 2 are typically universities and other scientific institutes, which can store sufficient data
and provide adequate computing power for specific analysis tasks. They handle analysis
requirements and proportional share of simulated event production and reconstruction. There
are currently around 160 Tier 2 sites covering most of the globe.
● Tier 3 corresponds to computing resources, which can consist of local clusters in a University
Department or even just an individual PC. There is no formal engagement between WLCG and
Tier 3 resources.
The most effective systems here are Tier-1 clusters, which are represented by well-organized
and powerful computing systems of participating national organizations and countries. In order to test
the performance of National Research Center “Kurchatov Institute” Tier-1 cluster (ANALY_RRC-
KI-T1) and compare it with other WLCG sites, 50k of 2-jet ggF Higgs boson production events were
generated with aMC_NLO Monte Carlo generator [15]. The results of this simulation are indicated in
Table 2.
Table 2. Parameters of 2-jet ggF Higgs boson Monte Carlo production at several WLCG sites
CPU Time, HS06 seconds
WLCG Site nEventsPerJob nFiles
Expected Total
ANALY_RRC-KI-T1 5000 10 28500000 17197806
ANALY_DESY-HH 5000 10 19550000 11357616
ANALY_INFN-LECCE 5000 10 17648874 10350000
ANALY_GLASGOW_SL6 5000 10 15050000 9020417
ANALY_IN2P3-CC 5000 10 15250000 4677853
ANALY_TOKYO_ARC 5000 10 14000000 9457170
The results of performance tests can vary depending on the condition of each individual
machine, its occupancy and some other parameters. However, the performance of considered GRID
sites are close to each other. Similar expected times mean that clusters offer similar compute power
as estimated by their specifications and certain benchmarks, while similar total times means that the
ratio of actual to expected performance is similar across all clusters.
27
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017
5. Conclusion
High performance computing systems are now an integral part of the high energy physics
landscape. One of the main physics fields where HPC has a major impact are the Higgs boson
studies. Increasing precision of measurements in particle physics leads to a complexity of subsidiary
calculations and HPC is a great resource to work with. It is also important to notice that some
analyses require even more precision. They have to use next-to-next-to-leading order corrections
(NNLO). Thus, the complexity of calculations will continue to increase in the near future.
In this paper, a brief overview of the high performance computing systems was presented. An
overview of the Titan supercomputer was provided, and its capabilities for the Higgs boson physics
was demonstrated. Performance of NRC-KI Computing cluster was studied and comparison with
other ATLAS GRID sites was shown. This comparison demonstrates that NRC-KI Computing
cluster is capable of handling tasks in high energy physics computing on par with other ATLAS
GRID sites.
Acknowledgement
We wish to thank all our colleagues who contributed to Monte Carlo simulation activities and
to PanDA software development and operations. This work was funded in part by the Russian
Ministry of Science and Education under Contract No. 14.Z50.31.0024 and the U. S. Department of
Energy, Office of Science, High Energy Physics and Advanced Scientific Computing under
Contracts No. DE-AC02-98CH10886 and DE-AC02- 06CH11357. The work is also has been carried
out using computing resources of the federal collective usage center Complex for Simulation and
Data Processing for Mega-science Facilities at NRC “Kurchatov Institute”, http://ckp.nrcki.ru/. The
work of R. Konoplich is partially supported by the US National Science Foundation under Grant
No.PHY-1402964. The work of K. Prokofiev is partially supported by a grant from the Research
Grant Council of the Hong Kong Special Administrative Region, China (Project Nos.
CUHK4/CRF/13G). This research used resources of the Oak Ridge Leadership Computing Facility at
the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S.
Department of Energy under Contract No. DE- AC05-00OR22725.
References
[1] Aad G. et al. [ATLAS Collaboration]. ATLAS Collaboration Observation of a new particle in the
search for the Standard Model Higgs boson with the ATLAS detector at the LHC // Phys. Lett. B
2012. V. 716 P. 1
[2] Chatrchyan S. et al. [CMS Collaboration]. Observation of a new boson at a mass of 125 GeV
with the CMS experiment at the LHC // Phys. Lett. B 2012. V. 716 P. 30
[3] Aaij R. et al. [LHCb Collaboration]. Observation of J/ψp Resonances Consistent with
Pentaquark States in Λ0b→J/ψK−pDecays // Phys. Rev. Lett. 2015. V. 115 P. 072001
[4] Aaij R. et al. [LHCb Collaboration]. Measurement of matter-antimatter differences in beauty
baryon decays // Nature Physics 2017. V. 13 P. 391
[5] Aad G. et al. [ATLAS Collaboration]. The ATLAS Experiment at the CERN Large Hadron
Collider // JINST 2008. V. 3 P. S08003
[6] Aad G. et al. [ATLAS Collaboration]. The ATLAS Simulation Infrastructure // Eur. Phys. J. C
2010. V. 70 P. 823
[7] URL: http://dashb-atlas-
job.cern.ch/dashboard/request.py/dailysummary#button=cpuconsumption&sites%5B%5D=All+Sites
&sitesCat%5B%5D=All+Countries&resourcetype=All&sitesSort=0&sitesCatSort=0&start=2017-01-
01&end=2017-09-
01&timerange=daily&granularity=Daily&generic=2&sortby=10&series=All&activities%5B%5D=D
28
Proceedings of the XXVI International Symposium on Nuclear Electronics & Computing (NEC’2017)
Becici, Budva, Montenegro, September 25 - 29, 2017
ata+Processing&activities%5B%5D=Group+Production&activities%5B%5D=MC+Event+Generatio
n&activities%5B%5D=MC+Reconstruction&activities%5B%5D=MC+Simulation&activities%5B%
5D=MC+Simulation+Fast&activities%5B%5D=MC+Simulation+Full&activities%5B%5D=Others&
activities%5B%5D=Testing&activities%5B%5D=Validation
[8] Skands P. Introduction to QCD // CERN-PH-TH-2012-196, DOI:10.1142/9789814525220_0008
[9] Deutschmann N. et. al. Gluon-fusion Higgs production in the Standard Model Effective Field
Theory // CERN-TH-2017-165 / NIKHEF-2017-035
[10] Bird I. et. al. Update of the Computing Models of the WLCG and the LHC Experiments //
CERN-LHCC-2014-014 / LCG-TDR-002
[11] URL: https://www.olcf.ornl.gov/computing-resources/titan-cray-xk7/
[12] Barner S. et al. Hardware and Software: Verification and Testing // 6th International Haifa
Verification Conference, HVC 2010, Haifa, Israel, October 4-7, 2010
[13] Michelotto M. A comparison of HEP code with SPEC benchmark on multicore worker nodes
// J. Phys.: Conf. Ser. 2010 V. 219 P. 052009
[14] Aad G. et al. [ATLAS Collaboration]. ATLAS Collaboration Prospective results for vector-
boson fusion-mediated Higgs-boson searches in the four lepton final state at the High Luminosity
Large Hadron Collider // ATL-PHYS-PUB-2016-008
[15] Alwall J. et al. MadGraph 5: Going Beyond // JHEP 2011 V. 06 P. 128
29