The evolution of the computing facilities and the way storage will be organized and consolidated will play a key role in how this possible shortage of resources will be addressed by the LHC experiments. The need for an effective distributed data storage has been identified as fundamental from the beginning of LHC, and this topic has become particularly vital in the light of the preparation for the HL-LHC run. WLCG has started an R&D within DOMA project and in this contribution we will report the recent results related to the Russian federated data storage systems configuration and testing. We will describe different system configurations and various approaches to test data storage federation. We are considering EOS and dCache storage systems as a backbone software for data federation and xCache for data caching. We'll also report about synthetic tests and experiments specific tests developed by ATLAS and ALICE for federated storage prototype in Russia. Data Lake project has been launched in Russian Federation in 2019 to set up a National Data Lake prototype for HENP and to consolidate geographically distributed data storage systems connected by fast network with low latency, we will report the project objectives and status.
The High Luminosity LHC (HL-LHC) will be a multi-Exabyte challenge where the envisaged Storage and Compute needs are a factor 10–100 above the expected technology evolution and flat funding (fig.1).
The WLCG community needs to evolve current computing and data organization models in order to introduce changes in the way it uses and manages the infrastructure, focused on optimizations to bring performance and efficiency not forgetting simplification of operations. These are the ingredients that will allow to drive down costs and be able to satisfy HL-LHC requirements. Technologies that will address the HL-LHC computing challenges may be applicable for other scientific communities (SKA, DUNE, LSST, BELLE-II, JUNO, NICA, etc.) to manage large-scale data volumes. The evolution of the computing facilities and the way storage will be organized and consolidated will play a key role in how this possible shortage of resources will be addressed by the LHC experiments. The need for an effective distributed data storage has been identified as fundamental from the beginning of LHC, and this topic has become particularly vital in the light of the preparation for the HL-LHC run. WLCG has started several R&Ds within Data Organization and Management Project (DOMA), one of which is a Data Lake project.
Data Lake is a set of sites, associated by proximity, providing together storage services, possibly accompanied by compute nodes to an identified set of user communities, capable to carry out independently well-defined tasks. Proximity could be defined by geography, connectivity, funding or a shared user community. This requires that their combined storage capacity and network bandwidth can meet the demands of the designated task and that usage of the different sites is transparent to the users, which, in turn, implies some form of trust relationship between the sites and a way to locate data, ranging from a simple file catalogue to a full-fledged namespace.
While access for users is transparent, the population and management of the storages within the Data Lake is a planned and managed activity. This includes the transitions between Quality-ofService (QoS) levels. These operations are done on the granularity of the Data Lake. Data is moved to or from the Data Lake as a whole, not to or from a specific site. Resource management within the Data Lake is the responsibility of the Data Lake.
Taking the aforementioned into account we can come up with some basic but crucial requirements for the future WLCG sites data storage infrastructure: Common namespace and interoperability. Coexistence of different QoS. Geo-awareness. Fault tolerance through redundancy of key components.
Scalability, with the ability to change the topology without stopping the entire system. Security with mutual authentication and authorization for data and metadata access. Optimal data transfer routing, providing the user direct access to the closest data location. Universality, which implies validity for a wide range of research projects of various sizes, including, but not limited to the LHC experiments.
In 2015 in the framework of the Laboratory "BigData Technologies for mega-science class
projects" at NRC "Kurchatov Institute" a work has begun on the creation of a united disk resource
federation for geographically distributed data centres, located in Moscow, Saint Petersburg, Dubna,
Gatchina (all above centres are part of the Russian Data Intensive Grid (RDIG) of WLCG) and
Geneva, its integration with existing computing resources and provision of access to these resources
for applications running on both supercomputers and high throughput distributed computing systems
(Grid) [
Data Lake for Scientific Data project supported by the Russian Science Foundation award has been launched in Russian Federation in 2019 to set up a National Data Lake prototype for HENP and to consolidate geographically distributed data storage systems connected by fast networks with low latency. JINR, SPbSU, PNPI, and MEPhI groups are participating at the first stage of the project and it is anticipated that more centres will be involved during the subsequent stages. The short-term plans for building a distributed Data Lake system in Russian Federation are shown on fig. 2.
We are considering EOS and dCache storage systems as a backbone software for data federation and xCache for data caching. Synthetic tests and experiments specific tests have been developed by ATLAS and ALICE for federated storage prototype in Russian Federation. site CE xCache site CE xCache site CE
xCache JINR SE dCache
EOS pools site CE xCache site CE xCache site CE xCache
The Computing Element (CE) + xCache computing infrastructure has been set up at PNPI and access to it has been configured from JINR. Sites’ technical characteristics are shown below: Worker Node @ JINR: 8 cores, Xeon E5420, 16GB RAM, 8.74 HEP-SPEC06 per Core Worker Node @ PNPI: 8 cores, Xeon E5-2680, 32GB RAM (VM), ~11 HEP-SPEC06 per
Core Local network @ JINR (SE ⇄ CE) 1 Gbps Local network @ PNPI (SE ⇄ CE) 10 Gbps Network IPv4,6 JINR → PNPI: Latency ~5 ms Network IPv4,6 PNPI → JINR: Latency ~10 ms Network IPv4,6 JINR → PNPI: Throughput ~1 Gbps Network IPv4,6 PNPI → JINR: Throughput ~1,5 Gbps
The following authorization parameters were configured and tested:
PNPI xCache → JINR SE: GSI authorization by local gridmapfile on JINR SE
PNPI WN → PNPI xCache: GSI authorization by VOMS (ALICE & ATLAS)
PNPI UI → JINR CE, PNPI CE (for local tests): GSI authorization by VOMS (ALICE &
ATLAS)
Hammer Cloud → ALL: GSI authorization by VOMS (ATLAS)
An external library for VOMS authorization in xCache [
The following tests were conducted during the infrastructure set-up phase: 1. Synthetic tests from Worker Nodes and through Cream-CE 2. ATLAS HammerCloud tests: a. Copy2scratch: data copy from WN to scratch area b. Directaccess: remote data access
Tests were conducted on 3 configurations: 1. Direct access from PNPI WN to JINR SE: “PNPI direct” 2. Access through xCache from PNPI WN to JINR SE: “PNPI xCache” 3. Local access from JINR WN to JINR SE: “JINR local”
Synthetic tests are composed of sequential transfers of identical files accessing the storage in three different ways: directly from the remote site, through xCache at the remote site (PNPI) and directly from the site local to the storage (JINR). Synthetic test results are as follows: For “PNPI direct” the speed was 650±40 Mbps (fig. 3-1).
EOS pools
JINR SE EOS mgm EOS pools site CE
xCache
EOS pools For “PNPI xCache” the speed was 6700±700 Mbps, but if we remove the first hit (cache warm-up) we will get the same speed with a much lower deviation. As a result, with 100 hits we have a 95% transfer speed gain and 92% transfer time gain (fig. 3-2). For xCache transfers the first warm-up access to a new file is highlighted with a red circle.
For “JINR local” with a 1 Gbps internal network tests show an expected transfer speed of 670±220 Mbps with a pretty high deviation (fig. 3-3).
The tests have been conducted for two days and results are presented in table 1. We can estimate a file transfer speed for Copy2Sctartch tests by “Download input files” metric, and for directaccess tests by “Athena Run Time” metric. Results of HammerClouds tests demonstrate 30% gain in time for copy2scratch and 12% gain in time for directaccess.
All of the components in the aforementioned tests were monitored by various monitoring
systems. Synthetic tests were run through WLCG middleware (CREAM-CE) and tasks execution
progress was monitored directly by CREAM-CE logs and monitoring tools. Tasks that were launched
using HammerCloud were tracked using HammerCloud and BigPanda monitoring [
Data Lake for Scientific Data is a three years R&D project started in Russian Federation in 2019 as a continuation of the successful Federated Data Storage project. Production-grade computing resources and 10–100 Gbps network connectivity with low latency is used to prototype a production Data Lake. The project will have two phases: 1. Four Russian WLCG centres (JINR, PNPI, SPbSU and MEPhI) will participate and they will be configured to test an initial prototype. The test methodology and consolidated monitoring tools should be developed, as well as we need a better understanding of xCache data access control scenarios. 2. More RDIG centres will participate to have a realistic set of production resources and scattered storages. This prototype will be tested for real LHC use-cases, monitoring tools will be provided to sites and experiments.
The Russian Data Lake R&D project is funded by the Russian Science Foundation under contract No.19-71-30008 (research is conducted in Plekhanov Russian University of Economics). The work at PNPI is supported by the NRC "Kurchatov Institute" order No. 1571.