The Data Knowledge Base (DKB) project aims at knowledge acquisition and metadata integration. It provides fast response for a variety of complicated queries, such as summary reports and monitoring tasks (aggregation queries) and multi-system join queries. Such queries are not easy to implement in a timely manner and, obviously, are less efficient than a query to a single system with integrated and pre-processed information would be. This work describes the status of the project as well as its integration with the ATLAS Workflow Management and future perspectives.
The Data Knowledge Base (DKB) project aims at knowledge acquisition and metadata
integration [
ETL (Extract, Transform, Load) pipeline flow [4] based on scripts and library;
System to run and check the flow; NoSQL database to store results; REST API to access system;
Frontend UI for users.
This system is used in the production system at ATLAS experiment to operate with GRID computing metadata and to prepare LCH Run 3.
DKB project has a distributed environment over several virtual machines hosted by CERN openstack infrastructure [5]. These machines are managed by computing center virtual machine software management system which includes Puppet and Foreman profiles. The whole environment is split over production, quality assurance and development servers. CentOS7 x86_64 operating system is used as base OS for all services. Production system is shown on figure 1.
es.atlas-dkb.cern.ch
api.atlas-dkb.cern.ch
ES aiatlas171 master
ES aiatlas172
It consists of two servers aiatlas171(master) and aiatlas172 with load balanced names assigned to them es.atlas-dkb.cern.ch and api.atlas-dkb.cern.ch accordingly. Elasticsearch [6] engine is used for data preservation and it is configured for two-copy replication mode. Such mode allows to achieve a good speed for read access and safety for data. There are two nginx servers used for system access to DKB from outside. First one works as proxy to ensure direct access levels to Elasticsearch engine for users with read-only or read-write permissions. Second one works as http server for DKB API software based on python FastCGI program. The main DKB workflow pipeline is configured to run only on the master node leaving slave node only for serving API requests.
The project sources are available on github [
API
api aiatlas172
Tag release data4es-prod
aiatlas171 https://github.com/PanDAWMS/dkb
Pull request with review master
New branch
After a new functionality or bug fix are added and tested in a new git branch, a new pull request is created for merging changes to the master branch. All pull requests go through careful review from another person in the project and only after that merges into master. Master branch automatically applies to the API server (based on puppet profile) and manually gets tags and applies as data4es-prod branch on the production workflow server. For the moment DKB provides API with version 0.3.3 and DKB production workflow runs version 0.2-0.
Current environment stores near 15GB data for ATLAS production tasks and 50GB of data for ATLAS analysis tasks. Every hour it loads and stores metadata information about around 1500 tasks and 5000 datasets from ATLAS experiment.
At present DKB serves for ATLAS collaboration Production System [
Information updates are based on “task timestamp” from ProdSys database. Main information comes from DEFT (Database Engine for Tasks) and is extended with additional metadata from other systems like:
JEDI - Job Execution and Definition Interface;
Rucio - scientific data management system.
At the end as soon as the new integrated metadata stored in the single Elasticsearh it simplifies search queries for the whole systems and such queries integrated into the ProdSys user interface through web access [6].
From implementation point of view, this workflow pipeline presents ETL process which is shown on figure 4. It is implemented through one Linux bash script calling different stages (workflow parts). These stages could use any software inside but to simplify communications between them and simplify building of such stages DKB python library is used.
Several changes have been made to the DKB project recently, mainly aimed at improving system performance and upgrading it to use a new version of the Elasticsearch engine. To improve the performance of user operations, a new metadata indexing model is implemented for ATLAS integrated metadata. It takes into account the specifics of the already addressed use-cases, and the most noticeable change is that the output datasets properties are now stored together with the Task object, in the form of nested documents (instead of parent/child documents). It is made to simplify queries to the Elasticsearch index, used in the most problematic requests from the addressed use-cases. Some investigations are made on internal communication protocol for DKB stages to use batch processing instead of serial one which is in place in production. The nearest plan for DKB is to fully migrate to the CERN production Elasticsearch infrastructure and split data storage from the project to special dedicated outside service [fig. 5].
The Data Knowledge Base project is successfully integrated with the Production System of the ATLAS experiment and it allows execution of complex analytical requests, requiring information from different information systems and from different levels of abstraction in a timely manner. The developed library and resent changes allows implementation of multiple different scenarios for metadata integrations, providing flexible tool for building metadata workflow pipelines. Stable run in production and good availability and accessibility allowed to use DKB metadata integration service in processing ATLAS metadata for tasks and datasets to prepare LHC Run 3.
The DKB ATLAS metadata integration services is supported by CERN IT and by CERN ATLAS IT support team. The UI for DKB added by ATLAS ProdSys development team by M. Borodin.
DKB project is supported by NRC "Kurchatov Institute".
Special thanks to M. Golosova and V. Aulov for the project development. experiments [DKB].
[CERN
[ProdSys].
at: