Currently-running large-scale scientific projects involve unprecedented amounts of data and computing power. For example, the ATLAS experiment at the Large Hadron Collider (LHC) has collected 140 PB of data over the course of Run 1 and this value kept increasing at the rate of ~800MB/s during Run 2. Processing and analysis of such amounts of data requires development of complex operational workflow and payload management systems along with building top edge computing facilities. In the ATLAS experiment a key element of the payload management is the Production and Distributed Analysis system (PanDA). It consists of several components and one of them is the BigPanDA monitoring component. It is responsible for providing a comprehensive and coherent view of the tasks and jobs executed by the system, from high level summaries to detailed drill-down job diagnostics. The BigPanDA monitoring has been in production since the mid-2014 and it continuously evolves to satisfy increasing demands in functionality and growing payload scales. Today it effectively keeps track of more than 2 million jobs per day distributed over 170 computing centers worldwide in the largest instance of the BigPanDA monitoring: the ATLAS experiment. In this paper we describe the monitoring architecture and its principal features.
The BigPanDA monitoring system is one of the PanDA workload management system [
Originally the monitoring system was developed for the ATLAS experiment [
An event refers to a distinct particle collision event recorded by the detector or simulated by a Monte Carlo software. A job is a payload which is supposed to process number of input events or produce them using initial random generator seeds conditions. A task is a collection of jobs united by the same data sample split along them. A request can contain a set or a chain of tasks as well as a single task. A campaign is a set of requests which are united by a physics objective. This hierarchy of objects has the following statistics on average: a job contains hundreds of events, a task contains hundreds of thousands of events, a request contains more than a billion of events and finally a campaign has a trillion of events. Similarly, the average time to process an event varies from seconds to several minutes, a job lasts from few hours to a couple of days, a task can take up to several weeks, a request is processed for more than a month and a campaign generally lasts for a year. These estimates show that the architecture of the monitoring system should be adequate to analyse and represent an extremely wide scale of data.
The BigPanDA monitor is built as a web application. The architecture of the system is shown
in Figure 1. The application backend is based on the Django model-view-controller framework [
Data-flow for the system is shown in Figure 2. The raw data from PanDA system is stored in the DB. Taking into account the fact that user requests may involve millions of DB rows to process, data aggregation algorithms are split between DB and Web server backends, which allows to reap benefits from both engines, reduce data transfers and increase the performance. In addition, the monitoring system has an advanced caching system. A data prepared for displaying is divided into a common and a user specific parts. In the common part data is proactively cached, whereas user specific data is processed for each user request individually. Pre-aggregated data from DB or cache storage is loaded through the standard Django interface as well as indexed data from ES. The user specific data can contain either settings for a page or a list of references to the most relevant pages determined by analysis of BigPanDA browsing history. The user specific data is protected by several policies, in particular, SSO authentication and HTTPS protocol.
Data is displayed using Foundation CSS [
The principal views of the monitoring system displays key objects of data processing and
analysis, such as jobs, tasks, files, and its aggregates. These views are unified into a single module
defining the system’s core (see Figure 3). More specific and accounting views are implemented as
specific plugins. This approach enables system customization by plugging in and out existing
components or implementing new ones. For example, the basic BigPanDA monitor core is installed on
Amazon Elastic Compute Cloud (EC2) [
The BigPanDA monitoring system is in production since the middle of 2014 and thanks to the flexible architecture the functionality of the system is continuously evolving without interrupting the service. In September 2018 the BigPanDA monitoring system in ATLAS handled more than 35 thousand requests in a day, where 77% of them are key views, in particular, jobs, tasks, sites, and files. The successful ATLAS experience made this product also of interest to other experiments, in particular at the moment of writing 3 instances of BigPanDA monitor serve payload monitoring for ATLAS, COMPASS and other experiments beyond High Energy Physics.
TPU team work is supported by the Russian Science Foundation grant under contract №16-11-10280.