Higgs boson physics is one of the most important and promising fields of study in modern high energy physics. To perform precision measurements of the Higgs boson properties, the use of fast and efficient instruments of Monte Carlo event simulation is required. Due to the increasing amount of data and to the growing complexity of the simulation software tools, the computing resources currently available for Monte Carlo simulation on the Large Hadron Collider (LHC) Grid are not sufficient. One of the possibilities to address this shortfall of computing resources is the usage of institutes' computer clusters, commercial computing resources and supercomputers. In this paper, a brief description of the Higgs boson physics, Monte Carlo generation and event simulation techniques are presented. A description of modern high performance computing systems and tests of their performance are also discussed. These studies have been performed on the Worldwide LHC Computing Grid and Kurchatov Institute Data Processing Center, including Tier-1 WLCG sites and the OLCF Titan supercomputer. Monte Carlo simulated events produced with the Titan supercomputer were used in the Higgs boson analysis and the results have been published by the ATLAS collaboration.
The observation of a new particle compatible with the Standard Model (SM) Higgs boson by the ATLAS and the CMS experiments [1, 2] has been an important step towards the validation of our understanding of nature. This observation is based on the information from the Large Hadron Collider (LHC). Experimental data was obtained during the first LHC data taking period (Run 1) in 2011-2012. Other impressive discoveries based on the LHC data include the pentaquark observation [3] and the observation of CP violation in rare baryon decays [4], both by the LHCb experiment.
To perform more detailed studies in these areas and to probe the new physics beyond the Standard Model (BSM), theoretical predictions for the physics processes under consideration must be obtained. The most convenient way to obtain such predictions is to use Monte Carlo (MC)-generated events. MC simulations are based on concrete theoretical models, which describe the processes under consideration.
MC simulations usually require a large amount of computing resources. Generally, there are two reasons for that. First, modern particle physics theories become more and more complex: they are based on non-trivial mathematics and calculations required by these theories are resourceintensive. Second, as far as measurements in particle physics are performed by the particle detectors, the detector effects must be also taken into account. The ATLAS detector [5] consists of millions of structural elements (more than 150M sensors). The response of each of them must be modelled and taken into account. For the physics analyses, it is necessary to perform so-called Full Simulation of the detector, which takes into account all the detector effects [6].
The resource intensity of MC tasks can be illustrated in terms of statistics. Figure 1 shows the fraction of CPU time consumption of all jobs submitted to Worldwide LHC Computing Grid (WLCG) by all its users during the first 35 weeks of 2017. The jobs were categorized according to their type: Data processing, Validation, Testing, MC Simulation, MC Simulation Full, MC Simulation Fast, MC Event Generation, MC Reconstruction, Group Production and Others. This plot shows that MC-related jobs take more than 85% of all CPU time provided by the WLCG.
Higgs boson physics is one of the most promising fields of study in modern particle physics. However, this topic is not only promising, but also complex. The most intensive channel of the Higgs boson production at the LHC in high energy proton-proton collisions is the gluon-fusion production (ggF). The Feynman diagram of such a process at the leading order (LO, first-order terms of Lagrangian in the Perturbation theory of Quantum Chromodynamics [8]) is shown in Figure 2.
When the center-of-mass energy of two protons is about 10 teraelectronvolts (TeV) or higher, more than 90% of all Higgs bosons are produced by the ggF mechanism. The process shown in Figure 2 is relatively simple, because there are no outcoming particles except the Higgs boson and there are no hadron jets (so-called 0-jet final state).
Considering second-order terms of the Lagrangian (Next-to-leading order corrections, NLO), the situation becomes more complex. Figure 3 presents some of the Feynman diagrams of ggF Higgs boson productions in the NLO.
The complexity of NLO calculations is directly correlated with the number of jets. In NLO
case, 0-jet category consists of 29 Feynman diagrams. The generation time of 100k events at CERN
on LXPLUS machines [
High Performance Computers (HPC) are presently the most valuable instruments for
CPUintensive tasks, such as Higgs boson production simulation with NLO corrections. One of the most
impressive realizations of supercomputer-based HPC system for science tasks, including high energy
physics, is the OLCF Titan, launched in 2012 by Oak Ridge National Laboratory, the performance of
which is higher than the computing power of all the ATLAS resources in WLCG [
The Titan supercomputer was used to perform the simulation of the Higgs boson production
associated with two hadron jets at NLO level with consequent decay of the Higgs boson to four
leptons: → → 4ℓ . Some technical parameters of this simulation are listed in Table 1,
namely: nFiles is the number of files in the input dataset; nEventsPerInputFile is the number of
events per input file and nEventsPerJob is the number of events per job. Expected CPU time is
estimated by internal GRID benchmarks based on previous information about the occupancy of the
particular GRID resource [
It is important to note, that while with CERN LXPLUS machines the generation event rate is about 2.5k events/hour, with the Titan it is about 650k/hour.
nEventsPerInputFile 2000
The datasets produced with the Titan supercomputer were subsequently used by the ATLAS
collaboration for physics analysis, with the results of this study presented in the public note [
The resource-intensive tasks can be handled not only by supercomputers, but also by WLCG computing facilities and cloud computing. WLCG is made up of four layers, or "tiers"; 0, 1, 2 and 3. Each tier provides a specific set of services: ● ● ● ●
Tier 0 corresponds to the CERN Data Centre, which is located in Geneva, Switzerland and also at the Wigner Research Centre for Physics in Budapest, Hungary. The two sites are connected by two dedicated 100 Gbit/s data links. All data from the LHC passes through the central CERN hub, but CERN provides less than 20% of the total computer capacity. Tier 0 is responsible for the safe-keeping of the raw data (first copy), first pass reconstruction, distribution of raw data and reconstruction output to the Tier 1s, and reprocessing of data during LHC down-times.
Tier 1 corresponds to thirteen large computer centres with sufficient storage capacity and with round-the-clock support for the Grid. They are responsible for the safe-keeping of a proportional share of raw and reconstructed data, large-scale reprocessing and safe-keeping of corresponding output, distribution of data to Tier 2s and safe-keeping of a share of simulated data produced at these Tier 2s.
Tier 2 are typically universities and other scientific institutes, which can store sufficient data and provide adequate computing power for specific analysis tasks. They handle analysis requirements and proportional share of simulated event production and reconstruction. There are currently around 160 Tier 2 sites covering most of the globe.
Tier 3 corresponds to computing resources, which can consist of local clusters in a University Department or even just an individual PC. There is no formal engagement between WLCG and Tier 3 resources.
The most effective systems here are Tier-1 clusters, which are represented by well-organized
and powerful computing systems of participating national organizations and countries. In order to test
the performance of National Research Center “Kurchatov Institute” Tier-1 cluster
(ANALY_RRCKI-T1) and compare it with other WLCG sites, 50k of 2-jet ggF Higgs boson production events were
generated with aMC_NLO Monte Carlo generator [
The results of performance tests can vary depending on the condition of each individual machine, its occupancy and some other parameters. However, the performance of considered GRID sites are close to each other. Similar expected times mean that clusters offer similar compute power as estimated by their specifications and certain benchmarks, while similar total times means that the ratio of actual to expected performance is similar across all clusters.
High performance computing systems are now an integral part of the high energy physics landscape. One of the main physics fields where HPC has a major impact are the Higgs boson studies. Increasing precision of measurements in particle physics leads to a complexity of subsidiary calculations and HPC is a great resource to work with. It is also important to notice that some analyses require even more precision. They have to use next-to-next-to-leading order corrections (NNLO). Thus, the complexity of calculations will continue to increase in the near future.
In this paper, a brief overview of the high performance computing systems was presented. An overview of the Titan supercomputer was provided, and its capabilities for the Higgs boson physics was demonstrated. Performance of NRC-KI Computing cluster was studied and comparison with other ATLAS GRID sites was shown. This comparison demonstrates that NRC-KI Computing cluster is capable of handling tasks in high energy physics computing on par with other ATLAS GRID sites.
We wish to thank all our colleagues who contributed to Monte Carlo simulation activities and to PanDA software development and operations. This work was funded in part by the Russian Ministry of Science and Education under Contract No. 14.Z50.31.0024 and the U. S. Department of Energy, Office of Science, High Energy Physics and Advanced Scientific Computing under Contracts No. DE-AC02-98CH10886 and DE-AC02- 06CH11357. The work is also has been carried out using computing resources of the federal collective usage center Complex for Simulation and Data Processing for Mega-science Facilities at NRC “Kurchatov Institute”, http://ckp.nrcki.ru/. The work of R. Konoplich is partially supported by the US National Science Foundation under Grant No.PHY-1402964. The work of K. Prokofiev is partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Nos. CUHK4/CRF/13G). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE- AC05-00OR22725. [1] Aad G. et al. [ATLAS Collaboration]. ATLAS Collaboration Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC // Phys. Lett. B 2012. V. 716 P. 1 [2] Chatrchyan S. et al. [CMS Collaboration]. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC // Phys. Lett. B 2012. V. 716 P. 30 [3] Aaij R. et al. [LHCb Collaboration]. Observation of J/ψp Resonances Consistent with Pentaquark States in Λ0b→J/ψK−pDecays // Phys. Rev. Lett. 2015. V. 115 P. 072001 [4] Aaij R. et al. [LHCb Collaboration]. Measurement of matter-antimatter differences in beauty baryon decays // Nature Physics 2017. V. 13 P. 391 [5] Aad G. et al. [ATLAS Collaboration]. The ATLAS Experiment at the CERN Large Hadron Collider // JINST 2008. V. 3 P. S08003 [6] Aad G. et al. [ATLAS Collaboration]. The ATLAS Simulation Infrastructure // Eur. Phys. J. C 2010. V. 70 P. 823