2010$ 2011$ 2012$ 2013$ 2014$ models simulating the expected behavior. In order to simulate the diversity of LHC physics, ATLAS produces more than 35 000 samples of simulated data per year. The variety of our simulated data adds value to the unified system for Big Data processing.

The LHC experiments employ the computational infrastructure of the Worldwide LHC Computing Grid (WLCG) [2]. The academic distributed computing environment of WLCG builds on the ideas proposed by Foster and Kesselman [3]. ATLAS leads in the WLCG usage in the number of data processing jobs and processed data volume. ATLAS manages more than 160 petabytes of data on more than hundred computational sites.

The central ATLAS production system (here referred to as ProdSys1) was designed and deployed more than nine years ago to address the needs of simulations and real data processing. Since the start of its operation, ProdSys1 proved itself as a valuable tool for processing real and simulated data on the Grid and became one of the enabling factors for the Collaboration.

ATLAS experience gained in managing simulations and data processing on an ever-increasing scale and handling scientific workflow requests of growing complexity led to a realization that ProdSys1 has to be significantly enhanced and updated. In this paper we highlight the production system techniques validated through practice, describe representative use cases handled by the production system and its upgrade – the ProdSys2.

10# supports dynamic sharing minimizing the workflow duration. Also, the production system enhances the resilience by managing failures in tasks and/or jobs, bulk job submission depending on success of the scout jobs, automatic e-mail notification, etc.

As the production workload increased in volume and complexity (the ATLAS production tasks count is above 1.6 million), the ATLAS experiment scaled up its data processing capabilities by upgrading the production system that unifies processing of the real and simulated data. To reduce the costs, the production system core remained shared by all use cases, despite the differences in their data processing requirements described in the next section.

The first step to new physics discoveries and high precision studies of rare events is the rejection of the benign events with “known” processes. The multi-tier trigger system reduces the exascale data flood to a petascale level by rejecting data from most of LHC collision events. In 2015, the ATLAS experiment will have a two-tier trigger system: 1. The hardware-based Level 1 trigger. 2. The software-based High-Level Trigger

designed by the ATLAS Collaboration [5].

The trigger data processing happens one step before the recording of “raw” data from the ATLAS detector.

Thus, any inefficiencies or mistakes may lead to unrecoverable loss of real data. To eliminate such losses, the dedicated trigger data processing workflow is employed to validate trigger software and other critical trigger changes during data taking. The trigger data processing is the main tool for commissioning the trigger for data taking. The challenge here is to achieve the fast turnaround, while avoiding data loss. The Trigger Data Real Data Simulated Data

Frequency

Weekly

Yearly Quarterly

Workflow

Length

Short Medium

Long

Number of Tasks

Tasks Duration

Data Loss

Several Hundreds Thousands

Day Weeks Months no no yes Fig. 3. Data processing workflow for the real data. Arrows are labeled with data transformation applications; boxes are labeled with various data types produced (see Appendix for data types descriptions).

Hard sca ering events Low-pT minimum bias events High-pT Minimum bias events

Event Genera on

Event Genera on

Event Genera on

Detector Simula on Detector Simula on Detector Simula on

Digi za on with pileup events

Trigger Simula on

Reconstruc on

Analysis Fig, 4. ATLAS simulations workflow is composed of many steps. Several initial steps are repeated for the hardscattering events and the minimum bias events (simulated in two complementary samples). experience with ProdSys1 shows that resubmission of the failed jobs delays the task completion. The resubmission technique has to be improved to meet the trigger data processing requirements.

The “raw” data from the ATLAS detector are processed to produce the reconstructed data for physics analysis. During reconstruction ATLAS applications process raw detector data to identify and reconstruct physics objects such as leptons. Figure 3 shows the data processing flow used in reconstruction.

The ATLAS collaboration completed four petascale data processing campaigns on the Grid, with up to 2 PB of real data being processed every year. Table 2 lists parameters for the ATLAS yearly data processing campaigns. (In 2013 reprocessing, 2.2 PB of input data were used for selecting about 15% of all events for reconstruction, thus reducing CPU resources vs. the 2012 reprocessing.)

Thanks to the quality of the ATLAS software and improvements in data processing workflow, in the 2011 data reprocessing only two collision events (out of 0.9 109 events total) could not be reconstructed due to subtle software bugs. Correcting these bugs allowed the Campaign year remaining events to be processed in a dedicated data recovery step.

In rare cases, the data corruption during job execution has been missed by the software or computing framework. This is called silent data corruption, which is detected later during data analysis. For example, the silent data corruption was detected in six events from the reprocessed 2010 data and in five adjacent events from the 2011 reprocessed data. Lowering the event losses below the 10-8 level enables recovery of the lost data on a case-by-case basis.

The computational resources required to process the simulated data dominate the overall resource usage. The data processing campaigns for the simulated data correspond to the data taking periods of the real data. The LHC data taking periods of the same conditions are characterized by the same center-of-mass energy, instantaneous luminosity, detector configuration, etc. Table 3 lists the major data processing campaigns for the simulated data.

The LHC instantaneous luminosities result in the presence of a large number of simultaneous collisions in the same event, overlapping the hard scattering event of interest. The presence of these additional minimum bias events is usually called “pileup”. To provide realistic simulation of these conditions, the data processing workflow for simulated data is composed of many steps (Figure 4): generate or configure hard-processes, hadronize signal and minimum-bias (pileup) events, simulate energy deposition in the ATLAS detector, digitize electronics response, simulate triggers, reconstruct data, transform the reconstructed data into data types for physics analysis, etc. The intermediate outputs are merged and/or filtered as necessary to optimize the chain.

An example of a more complex workflow used to simulate the ATLAS trigger using dedicated hardware for fast tracking (FTK) designed by the ATLAS Collaboration [6] is shown on Figure 5, where to keep the computational resources for the FTK simulation below practical limits, every event is split into 256 sub-regions [7]. In the three-step workflow, each event is processed by 64 jobs; each job simulates tracks in four FTK sub-regions one after another. The sub-region merging is done in two steps: producing whole regions, then whole events in the NTUP_FTK files. The final step uses FTK tracks in trigger simulations producing the reconstructed data in DESD_FTK files or adds FTK tracks to the simulated events in the RDO_FTK files (see Appendix for the data types descriptions).

The FTK workflow demonstrates the production system workflow with sub-events processing (vs. traditional processing of the whole events). We expect that next-generation processor technologies (such as many-core) will increase the number of workflows with fine-grained processing.

Validating the Big Data processing techniques, four different sub-campaigns of the mc11 campaign implemented the pileup conditions, detector conditions and geometry increasingly closer to those in real data. 2: Each FTK input event is processed by 64 jobs - one job per tower region 2a: For each of the 64 regions, simulate FTK tracks in each of the four sub-regions per region 1: Skim silicon data from each event into one FTK input event

RDO

NTUP_ FTKIP

RDO

2b: Merge all four sub-regions for every tower

region 3a: Merge all 64 tower regions into whole event reg0 … reg63 RDO

RDO NTUP_FTK

NTUP_FTK 3b: Reconstruct or merge each

event with corresponding merged FTK tracks

DESD_FTK RDO_FTK During the mc12 campaign, the majority of the events was simulated in the sub-campaign mc12b. Later, the mc12c sub-campaign implemented an improved detector geometry description. The goal of the mc14 campaign was to prepare for the 2015 data taking. The 8 TeV events were processed with improved and updated simulation, digitization and reconstruction software while using the same conditions as in the mc12 campaign. The 13 TeV campaign had the center of mass energy expected for the 2015 data taking with estimated pileup and detector conditions. The mc14 campaign used the new ATLAS Integrated Simulations Framework described in Ref. [8], with multicore processing becoming the default for major simulated data processing steps: simulation, digitization and reconstruction.

The ProdSys1 experience made clear that the new level of abstraction – the workflow – is necessary to scale up the system in support of the growing number of tasks and data transformations. The workflow is a group of interdependent tasks, where dependencies exist in the form of the input/output datasets. Such would be the case with a “chain” type simulations workflow (Figure 4), where the data goes through transformations with possible other steps like merging interspersed in between.

DEfT is the service that implements the workflow abstraction in the upgraded production system. Due to relative autonomy of DEfT, i.e. it’s agnostic approach to the computational resources used (which are in the scope of the JEDI component), it is appropriate to describe it as a state machine, in which the states of the workflow components undergo a series of transitions based on certain predefined rules and external events, such completion of a job managed by JEDI. By itself, DEfT is not a workload management or resource provisioning system. It is a higher-level abstraction layer encapsulating institutional knowledge that allows scientists to concentrate on the design, monitoring and effective management of production workflows.

The ProdSys1 experience demonstrated that it is difficult to generalize the procedure of job definition since the method used to define jobs depends on the task type. For improved maintainability and flexibility JEDI has a pluggable structure so that a plug-in defines jobs for each use case.

Dynamic job definition means that the job is defined at some point after the creation of its parent task, and taking into account operating conditions that were not known at the task inception. This is in contrast to a static definition of the job that happens immediately once the task is created. There are advantages realized once there is a capability to define jobs dynamically, based on the actual resources and other conditions present once the task moves into the execution stage. For example, the late binding takes advantage of defining small jobs for the sites with limited capabilities (slow CPU, low disk, network, etc.) or defining extremely large jobs for processing on the HPC facility with hundred thousand CPU-cores. Another advantage is resubmission of the long failed job redefined as several short jobs. This capability avoids delays in task completion for use cases such as trigger data processing described in Section 3.1.

We further upgraded the production system to improve the performance and accommodate a growing number of new requirements and use cases. Figure 7 shows that the upgraded system is deployed in production supporting complex workflows with tasks processing simulated and real data for future scientific Tasks Requests Layer: UI

Task Defini on Layer: DEfT

Job Defini on Layer: JEDI

Job Execu on Layer: PanDA