LHC Computing Grid was a pioneer integration effort, managed to unite computing and storage resources all over the world, thus making them available to experiments on the Large Hadron Collider. During decade of LHC computing, Grid software has learned to effectively utilise different types of computing resources, such as classic computing clusters, clouds and hyper power computers. While the resources experiments use are the same, data flow differs from experiment to experiment. A crucial part of each experiment computing is a production system, which describes logic and controls data processing of the experiment. COMPASS always relied on CERN facilities, and, when CERN, during hardware and software upgrade, started migration to resources, available only via Grid, faced the problem of insufficiency of resources to process data on. To make COMPASS data processing able to work via Grid, the development of the new production system has started. Key features of modern production system for COMPASS are: distributed data processing, support of different type of computing resources, support of arbitrary amount of computing sites. Build blocks for the production system are taken from achievements of LHC experiments, but logic of data processing is COMPASS-specific.
All physics experiments have same steps of data taking, processing, archiving, but details of these steps are absolutely different for each experiment. Implementation of data processing of the experiment comes from its computing model, which, in its turn, is described by physics processes, collected data type, volumes, chosen software technologies, data transformations, available computing resources and their types, type of storage, etc.
Data of COMPASS experiment [
Being resident or CERN, COMPASS depends on CERN IT services to store and process
data. During experiment’s lifecycle (data taking of the experiment has begun in 2002), some of IT
services became obsolete and during next 2-3 years the following services will be replaced by more
modern ones: Castor by EOS (link), lxbatch LSF by Condor, AFS by EOS. This process of
simultaneous replacement of computing infrastructure components strongly influences data
processing of the experiment and triggered changes of software components which interact with
computing site, data, conditions and metadata storage. In order to ease consequences of current and
future infrastructure changes, the computing model of the experiment was adapted accordingly
[
To manage tasks and jobs in such distributed infrastructure, special software must be created, because data processing of any experiment is unique. Usual name of such software is a production system. It covers all steps of data processing: from task definition and datasets selection, jobs submission and monitoring of their statuses, to decision making mechanisms, which control data processing through all the steps. In case of COMPASS such software was already presented, however, during the migration to Grid environment and distributed computing, it became clear that it has to be replaced by a brand new one. The reasons were the following:
- previous system was too strongly integrated with existing computing infrastructure and it was prepare to work in “local” environment of production manager’s account; - it was not designed to work with any other type of computing resource except LSF; - it used commands, available only in interactive mode to submit and control jobs. Adding one more computing site in the previously used production system was impossible.
Thus, the development of the new production system, which has to cover all the needs of data processing in the distributed heterogeneous computing environment and will be able to overcome the limitations of the previous implementation described above, has begun.
The production system of the experiment must meet the following expectations: - must support data processing from task definition till data archiving; - must provide support of all types of data processing: Monte-Carlo simulation, reconstruction, user analysis;
- must require minimum software development and include as much as possible components of already developed systems; - must support any type of computing and storage resources; - must provide user-friendly and fully functional interface; - must be secure, flexible, easy to extend and deploy; - must provide monitoring of each steps of data processing and user actions.
Production system has the following list of components:
- tasks and jobs definition and management interface;
- status tracking and decision making mechanism; - data management mechanism; - jobs submission mechanism; - jobs delivery to remote sites; - jobs execution on remote sites; - monitoring.
Since logic of processing of each experiment is unique, tasks and jobs definition, management interface and decision tracking mechanism had to be developed from scratch.
All other components of the infrastructure initially were developed to cover needs of ATLAS distributed computing and were adapted for COMPASS: PanDA workload management system manages jobs delivery, execution on remote sites and monitoring via its components: PanDA server, AutoPyFactory,
Figure 1. Task definition interface Pilot and Monitoring. Django framework was chosen to build web user interface. RDBMS is MySQL. Programming language is Python, it is the same for all components of the system, which makes it easy to integrate. Decision making mechanism organised as set of micro-services, each runs periodically and performs its small operation basing on conditions in system’s core database. Data management mechanism prepares, delivers and archives files.
Once components of the system work as independently, time-out, retry, load balancing mechanisms are implemented in order to achieve maximum use of available resources. Jobs submission performed automatically, amount of jobs to be submitted at each submission cycle is calculated basing on amount of running jobs, results of previous submissions, etc. Jobs submission and status checking in the system are divided in order to reduce load on services and machines where services are running.
Screenshots of production system user interface and services and jobs monitoring pages are shown on the figures 1-3.
Future plans include enabling new computing resources, and, also, new types of computing resources, such as HPC facilities, in particular Blue Waters HPC of University of Illinois at UrbanaChampaign, where collaboration has active allocation.
Monte-Carlo processing will be covered by the production system, at the moment MC production is done by Analysis coordinators and users groups in their home institutes as users analysis, separately and not organised into one managed effort.
Users analysis will also be moved under the production management system.
Another direction of development is integrating Rucio [
Serious integration and software development efforts have been performed during 2016-2017 in order to organise a distributed processing of data gathered by experiment. New system has been working in a production mode since August 2017. During this period, via new Grid Production System almost 2 millions chunks, collected by COMPASS during 2015-2017, were processed and 0.5PB of resulting data was generated. Processing rate handled by the system is ~12000 of simultaneously running jobs.
All steps of processing are covered with rich monitoring services, allowing to get full picture of data processing at any particular period of time.
COMPASS collaboration received a brand new production system, based on widely used and actively supported software components and secured itself against decommissioning of LSF and phase-out of AFS and Castor.
All management components of the system are deployed on the JINR cloud service [