Over the years, ATLAS has developed a large number of monitoring and accounting tools for distributed computing applications. In advance of the increased experiment data rates and monitoring data volumes foreseen for LHC Run 3, starting in 2012, a new infrastructure has been provided by the CERN-IT Monit group, based on InfluxDB as the data store and Grafana as the display environment. ATLAS is adapting and further developing its monitoring tools to use this infrastructure for data and workflow management monitoring and accounting dashboards, expanding the range of previous possibilities with the aim of achieving a single, simpler, environment for all monitoring applications. This contribution describes the tools used, the data flows for monitoring and accounting, the problems encountered and the solutions found.
Every experiment that produces and processes large amounts of data needs to monitor the
infrastructure and the applications that deal with these data. Monitoring is essential to be able to spot
and fix any system failure in a short time and to identify ways to improve the system performance,
within the available hardware resources. The ATLAS experiment [
The Distributed Data Management system Rucio [
ATLAS used during LHC Run 1 (2009–2012) and Run 2 (2015–2018) a monitoring and accounting infrastructure for the distributed computing applications developed about 10 years ago by CERN-IT together with ATLAS members. These “old dashboards” started showing aging effects in the last few years, visible primarily as an increasing slowness of data retrieval due to the massive amount of data in Oracle databases. Also, the lack of in-depth knowledge for maintenance as most original developers left long ago, led to a lack of flexibility and impossibility to develop new views and/or data correlations across different data sources. This system worked well enough for general monitoring until the end of Run 2 last year but was evidently in need of a good refurbishing.
Since 2016 the CERN-IT MonIT group started developing a new infrastructure and environment for monitoring and accounting applications based on modern Open Source components, and ATLAS started implementing “new” dashboards using this infrastructure, for data and workload accounting and global monitoring.
In the meantime, the BigPandaMon [
In recent years many Analytic tools have appeared on the market. They can be used for more detailed investigations and to correlate data from different sources. The Analytics cluster provided by the University of Chicago allows a more interactive use of monitoring data for detailed investigations and correlations between the various distributed computing systems.
The CERN-IT MonIT group provides “Monitoring as a Service” for the CERN data centre and
the WLCG collaborations. The services consist in providing the infrastructure to collect, transport,
process and store the monitoring data, and the dashboards to display all collected information. The
diagram in Figure 1 shows the components used and the data flow through them:
A number of data collector units receive information from the services to be monitored and feed
the data pipeline. The relevant collectors for our applications are the messaging system ActiveMQ
[
There are three groups of dashboards, with different read/write access parameters and frequencies of upgrades: the Production dashboards are stable and updated only with tested improvements; the Development dashboards are used to test new features or views before implementing them into production; the Playground dashboards are used to explore new possibilities or tools.
Several groups of dashboards have been developed to monitor ATLAS Distributed Computing (ADC) applications. For Distributed Data Management, data usage and transfer information is constantly sent by Rucio to the message brokers and then further processed; data storage information is periodically extracted from the Rucio database in Oracle, dumped to HDFS and further processed to be stored in InfluxDB. Many views are available, including historical views going back to the start of Rucio in 2015, current snapshots of data volumes by site or data type, data transfer rates and efficiencies between any two endpoints. Figure 2 shows two examples of DDM dashboards.
The second most important group of dashboards refer to job monitoring and accounting. In
this case the information is collected from the PanDA database every ten minutes and stored into
1hour time bins; 24-hour, 7-day and 30-day bins are calculated automatically. The dashboards display
the number of pending, running or finalising jobs, as well as statistics for the completed jobs including
errors, CPU and wall-clock time consumption, and many other job parameters. Information is also
imported from other sources, such as the site topology from the ATLAS Grid Information System
(AGIS) [
A dashboard for short-term but site-oriented job monitoring was also developed; here the data are aggregated by site, thus reducing the cardinality by a large factor (the number of sites, over 120), and keeping a reduced number of variables. In this way it is possible to display data directly in much smaller time bins and create automatic alarms in case of anomalous conditions in real time. An example of this dashboard is shown in Figure 4.
Several other dashboards have been developed to monitor the status of other central and distributed services and servers. In addition, a summary display of ADC monitoring dashboards was created: the “Live Page”. It is a web site with several pages that show some statistics and a few static plots extracted from the various dashboards and refreshed every hour; clicking on each plot leads to the relevant dashboard where more detailed investigations of any problem can be performed (see Figure 5). This is the entry point for shifters and computing managers wishing to have a global view of the current status of the ADC systems.
The MonIT dashboards provide a wealth of information but by design they contain only statistical aggregations of jobs and tasks, with their properties, stored as time series. For detailed monitoring of the processing of tasks and job status evolutions, the BigPandaMon application provides real-time access to the PanDA data in the Oracle database.
BigPandaMon is built as an aggregator of information from different sources, of which the PanDA database is the principal, but not the only, one. The retrieved information is cached for 10 minutes, allowing the users to digest it and formulate more detailed queries; if the new query only needs information that is already cached during the main query, the system will respond very fast, otherwise a new query to Oracle will be launched and the response time will depend on the amount of retrieved data – usually a few seconds will suffice. Figure 6 shows the data flow within BigPandaMon.
From the top page, users can select a number of views or search directly for a given task or job, and get all details, including the relations between tasks and jobs, sites where the jobs are running or have run, log files, error conditions and so on. Every displayed piece of information is clickable and leads to the source of this information and additional details. Figure 7 shows examples of the task and jobs tables, as well as the task-level statistics on job execution times, CPU and memory usage, and task completion rates.
The analytics cluster at the University of Chicago provides an interactive environment to
develop additional or alternative dashboards and investigate correlations between several data sources.
It is complementary to the monitoring and accounting infrastructure at CERN. In addition to the
information imported from the PanDA and Rucio databases at CERN, it collects data from the WLCG
File Transfer System (FTS) [
ATLAS Distributed Computing has a coherent set of monitoring and accounting dashboards and interactive tools. Technologies evolve all the time, so we have to follow them; we try to use Open Source solutions as much as possible, even if at times some home-made parts are inevitable, for example because of the low number of display options available in Grafana.
The future is in the more interactive environments providing the possibility to correlate information from many different sources. ATLAS has already started in this direction and is actively developing new views and new tools to increase the level of automatic error or anomaly detection in view of the start of LHC Run 3 in 2021.
This work was performed in the context of the ATLAS Collaboration. It was funded for the part related to workflow management monitoring by the Russian Science Foundation under contract No. 19-71-30008 (research is conducted in Plekhanov Russian University of Economics).