A substantial data volume growth will appear with the start of the HL-LHC era. It is not well covered by the current LHC computing model, even taking into account the hardware evolution. The WLCG DOMA project was established to provide data management and storage researches. National data lake r&d's, as a part of the DOMA project, should address the study of possible technology solutions for the organization of intelligent distributed federated storage. This talk will present the current status of the Russian Scientific Data lake prototype and the methodology, which is used for the validation and functional testing of deployed infrastructure.
High Luminosity LHC (HL-LHC) will be a multi-Exabyte challenge where the envisaged Storage and Compute needs are a factor 10 above the expected technology evolution. WLCG community needs to evolve current computing and data organization models in order to introduce changes in the way it uses and manages the infrastructure, focused on optimizations to bring performance and efficiency, not forgetting simplification of operations.
Technologies that will address the HL-LHC computing challenges may also be applicable to
other communities, such as SKA, DUNE, CTA, LSST, BELLE-II, JUNO, etc. and allow them to
manage large-scale data volumes. One of such technologies that we will discuss in this paper is Data
Lake [
Generally speaking, Data Lake is a set of sites, associated by proximity, providing together storage services, possibly accompanied by compute ones, to an identified set of user communities, capable to carry out independently well-defined tasks. Proximity could be defined by geography, connectivity, funding or a shared user community. This requires that their combined storage capacity and network bandwidth can meet the demands of the designated task and that the usage of different sites is transparent to the users. This implies some form of trust relationship between the sites and a way to orchestrate their shared resources.
In this work we will focus on a specific scenario of a national Data Lake with a common namespace, providing storage for a set of relatively small CPU-oriented sites alongside a large national research center with a reliable storage system [fig.1]. Our goal is to optimize the computational effectiveness by using techniques like read caching and write buffering, minimizing the CPU idle time during heavy data I/O operations.
A national Data Lake prototype was deployed on a number of Russian sites connected by a
modern network infrastructure also used for processing of LHC data (part of the LHC ONE network)
[fig. 2]. In our prototype JINR acts as a Data Lake entry point providing data storage and
orchestration. Three other sites (PNPI, MEPhI and PRUE) act as CPU-oriented resources with
relatively small volatile local storages used as caches.
The prototype is being built using the following building blocks:
Resources
o Bare-metal at JINR and MEPhI
o Virtualized at PNPI and PRUE
Storage systems
o EOS [
In order to conduct efficiency tests authors have used both synthetic and real-life tests. Synthetic tests were basically generating files of a given size with random contents and measuring read/write performance of these files in various conditions. Real life tests involved application from ATLAS software stack reading and processing event files. Different set of tests was used for read- and write-intensive scenarios.
The first synthetic tests were mostly designed as functional tests with an additional goal of measuring maximum "theoretical" efficiency in ideal conditions. The tests were reading a predefined set of files of a certain size with a certain number of repetitions. Three scenarios were examined: 1) No cache (reference value) 2) Dedicated cache on a single server 3) Distributed cache on the worker nodes
Caching scenarios were primarily targeted at workloads that reprocess data, i.e. read the same
input data file more than once. Such payloads are not uncommon as can be seen in the ATLAS data
popularity studies [
The next step was to test the real-life experiment payloads. This was achieved using the HammerCloud stress-testing system submitting ATLAS analysis tasks.
Write buffering is targeted at workloads that mostly write data. In order to optimize the CPU efficiency on the worker nodes, generated data is written into the local buffer and then migrated into the Data Lake in the background automatically by the storage system. Currently only synthetic tests are developed for this scenario. HammerCloud requires some modifications in order to be used for this kind of workloads. A fundamental agreement has been reached with the developers of this system, but the implementation is still a work in progress.
Unlike in caching tests, in this case it was decided not to measure the I/O speed directly, but to compare the CPU usage efficiency on the worker nodes. In order to do this one needs a real-life CPUintensive workload that will generate data and on top of that the CPU time it consumes must be significantly higher than the time that is necessary for saving the generated data on the local storage.
As with the caching tests, buffering must be examined using multiple scenarios: 1) No buffer (reference value) 2) Dedicated buffer on a single server 3) Distributed buffer on the worker nodes
Due to technical difficulties the third scenario has not been implemented yet.
For the monitoring of the Russian Data Lake prototype, a unified monitoring system was created using the ELK stack [fig. 3]. This stack includes ElasticSearch (non-relational data storage), Logstash (collector and data processor) and Kibana (data visualization system). In total, four dashboards were implemented in the Kibana system: 1. Xrootd dashboard based on the processed data from the event logs for monitoring the
Xrootd component 2. Billing monitoring dashboard to get information about dCache 3. Jobs monitoring provides information about all test jobs which run on the Russian
Data Lake prototype computing resources 4. Accounting dashboard containing the information about the consumed CPU resources Additionally, a web application was developed for monitoring and analytics of the Russian Data Lake prototype testing. It uses data from the same job monitoring index in the unified ElasticSearch storage and mitigates a lack of advanced charts customizing features in Kibana visualization platform providing ready-for-publication quality comparison plots.
The perfSONAR system is used separately to collect and analyze information about data transmission on the global network. The prototype structure assumes that each site has a dedicated perfSONAR node for network measurements.
This work was funded in part by the Russian Science Foundation under contract No.19-7130008 (research is conducted in Plekhanov Russian University of Economics)