Structured data storage technologies evolve very rapidly in the IT world. LHC experiments, and ATLAS in particular, try to select and use these technologies balancing the performance for a given set of use cases with the availability, ease of use and of getting support, and stability of the product. We definitely and definitively moved from the “one fits all” (or “all has to fit into one”) paradigm to choosing the best solution for each group of data and for the applications that use these data. This paper describes the solutions in use, or under study, for the ATLAS experiment and their selection process and performance measurements.
When software developments started for ATLAS [
Having only one underlying technology helped to provide a robust and performant central
database service, managed jointly by CERN at the system level and ATLAS at the application level.
Many time-critical applications are now hosted by the CERN Oracle infrastructure:
The conditions database (COOL) [
On the other hand having only one underlying technology forced some applications that have
no need of relational information into fixed schemas that may be not completely optimal; for
example time-series measurements produced by DCS (Detector Control System) can be more simply
represented by time-value pairs, and their data have to be compressed before storing in Oracle
because of their huge sizes. In addition Oracle schemas have to be carefully designed upfront and are
then hard to extend or modify, and data access to Oracle databases from Grid jobs was less than
obvious and an interface system (Frontier [
Towards the end of LHC Run1 in 2012 and during the shutdown period in 2013-2014 a
number of new structured data storage solutions ("NoSQL Databases") were tested as back-end
support systems for new applications, including Hadoop [
The database systems are used to support ATLAS data processing and analysis, as well as all other collaboration activities. One can identify three major groups of data: Conditions data. They are all non-event data that are useful to reconstruct events, such as detector hardware conditions (temperatures, currents, voltages, gas pressures and mixtures, etc), detector read-out conditions, detector calibrations and alignments, and physics calibrations. All conditions data have associated intervals of validity and (for derived data) versions. The COOL database is used for all conditions data. Physics metadata. This is information about datasets and data samples, provenance chains of processed data with links to production task configurations, cross-sections and configurations used for simulations, trigger and luminosity information for real and simulated data. Distributed computing data management and processing book-keeping. The distributed production/analysis and data management systems produce and need to store a wealth of metadata about the data that are processed and stored: o Rucio (Distributed Data Management) has a dataset contents catalogue (list of files, total size, ownership, provenance, lifetime, status etc.) a file catalogue (size, checksum, number of events), a dataset location catalogue (list of replicas for each dataset) and keeps information on the activities of data transfer tools, deletion tools and on storage resource status etc. o ProdSys/JEDI/PanDA (Distributed Workload Management) store lists of requested tasks and their input and output datasets, software versions, lists of jobs with status, running locations, lists of processing resources with their status etc.
Both systems use a combination of quasi-static and rapidly changing information, as ATLAS runs over 1 million jobs/day using on average almost 300k job slots and moves 600 TB/day around the world. Oracle supports very well both systems if the tables and the load don't grow indefinitely; “old" information is automatically copied to an archive Oracle database and removed from the primary one.
All this information is stored in three main Oracle RACs (Real Application Clusters), respectively for ATLAS online, offline, and distributed computing applications, plus an archive database, all with active stand-by replicas and back-ups. Selected users and processes have write access; all users have read access. Read access normally goes through front-end web services as direct access to Oracle from many processes could overload the servers: Frontier for access to conditions data from production and analysis jobs, the AMI and COMA front-end servers for access to metadata, and DDM and PanDA servers for access to dataset and production/analysis task information. Figure shows a sketch of the Oracle RACs and the data flow between them, including the replication to the active stand-by instances and the distribution of conditions data to IN2P3-CC in Lyon (France), RAL in Oxfordshire (United Kingdom) and TRIUMF in Vancouver (Canada).
The main distributed computing applications (Rucio and ProdSys/PanDA) have a very high transaction rate and the Oracle database is very efficient in dealing with this large information flow. Applications such as monitoring and accounting, that only read from the database, are instead better suited for different storage systems, with needed data extracted from Oracle and formatted appropriately for the expected queries. Tasks to extract the relevant information from Oracle and store it in Hadoop run continuously and provide input to several other tools, including dataset popularity, task monitoring and data management accounting.
ElasticSearch [
The ATLAS EventIndex [
The EventIndex has a partitioned architecture, following the data flow, sketched in Figure .
The Data Production component extracts event metadata from files produced at Tier-0 or on the Grid,
the Data Collection system [
At the time of writing the Hadoop system stores 120 TB of real data and 36 TB of simulated data, plus 154 TB of other data (input and transient data and archive). In Oracle we have over 100 billion event records, stored in a table of 2.2 TB with 2 TB of index space.
An active R&D programme to explore different, and possibly better performing, data store
formats in Hadoop was started in 2016. The "Pure HBase" approach (database organized in columns
of key-value pairs) was one of the original options in 2013, but did not work in 2015 because of the
then poor performance of the CERN lxhadoop cluster (problem solved at the end of 2015); it is more
promising now as it shows good performance for event picking. The Avro [
The continued usage of Oracle is fine for the time being but we were warned by CERN that the license conditions may change in the future, so some kind of diversification may be needed. Some types of data and metadata fit naturally into the relational database model, but other data much less, for example the large amounts of useful but static data on DDM datasets for accounting, or information on completed PanDA production and analysis tasks, event metadata and so on.
As long as access to the data is done through an interface server, the user won't actually see the underlying storage technology. In this way it is possible to keep only the "live" data in Oracle and move the rest to different technologies. This also means that at some point in the future we could change technology for the SQL database without too much trouble.
CREST [
The CREST system for ATLAS is under active development and will be in production for the start of LHC Run3. By that time all existing conditions for Run1 and Run2 will have to be transferred to the new system, to allow processing and analysis of all ATLAS data with the most recent software suites.
Time series are for example streams of DCS (Detector Control System) data, where for each data type a raw data record consists of a time stamp and one or a few values. This information is currently stored in Oracle using COOL, after averaging over short time periods, or storing new values only when sufficiently different from previous ones. Data sizes can become enormous compared to other data types, so much that direct use of this information in reconstruction jobs is not a good idea; it is much better to store this information in a system that is designed for time series and has useful tools for averaging over predefined time intervals, threshold detection, and an integrated display of the values as a function of time.
CERN-IT decided to use InfluxDB [
A new effort to revise and harmonize metadata information and its storage and retrieval tools started this year in ATLAS: the DCC (Data Characterization and Curation) project. It has three complementary approaches, respectively on the overall architecture, a top-down approach for dataset related metadata and dataset discovery based on the Data Knowledge Base, and a bottom-up approach to event metadata based on the concepts of the Event WhiteBoard and Virtual Datasets.
The Event WhiteBoard (EWB) project has been launched recently. It is an evolution of the EventIndex concept, but with an event-oriented architecture, whereas the EventIndex has a datasetoriented internal storage organization. It will have one and only one logical record per event, containing event identification and immutable information (trigger, luminosity block etc.), but then for each processing step involving each event it will have the link to the algorithm producing it (processing task configuration), pointers to outputs and flags for offline selections (derivations). An important new feature of the EWB is the possibility for automatic processes and single users to annotate single event records, adding key-value pairs (or similar formats) that can then be interpreted and used for automatic selections or other actions. Figure shows the EWB component architecture and the flow of data into and out of it.
The problem is not intrinsically difficult but the EWB will have to support 10 billion new real and 35 billion simulated events per year for Run2, a factor of 3 more for Run3 and another factor of 3 more for Run4. Work on the technology selection is starting now, with the aim to have a prototype working at the Run2 scale by the end of 2018 that will be promising to scale as required for Run3, and the new EWB in operation during 2020.
Virtual Datasets (VDS) are not a new idea but with the new EWB technology it should be possible to implement them. A VDS is a list of events that satisfy a number of conditions as contained in the EWB. For example, the derivation step after reconstruction now writes out O(100) streams with selected events; even if the events are “slimmed”, the amount of required disk space is large. With VDSs, it will be enough to flag the selected events in the EWB, saving lots of storage, and user analysis jobs will then read only those events.
ATLAS is always following technology developments in the database and structured data storage fields. The lifetime of ATLAS computing tools and infrastructure is much longer than the active lifetime of many open source products, and this fact poses very strong constraints on product selection. In any case we need to continue the R&D programs to make the best use of new upcoming computing technologies, without neglecting ongoing operations of course.
Continued collaboration with CERN-IT is essential for providing well-performing and robust services to the Collaboration.
The tool that is invisible to most users is the one that works without problems all the time!