DB Back-ended Filesystem for Science

Abstract

dCache is a distributed storage system developed at Deutsches Elektronen-Synchrotron (DESY) in collaboration with Fermi National Accelerator Laboratory and the Nordic eInfrastructure Collaboration (NeIC) to manage large numbers of disk servers and ensure transparent data migration to and from archival storage. Its multifaceted approach provides an integrated solution to support various scientific use cases using the same storage infrastructure, including high throughput data ingest, data sharing over wide area networks, eficient access from High-Performance Computing (HPC) clusters, and long-term data persistence on tertiary storage. The namespace/metadata component of dCache, known as Chimera, is built on top of a relational database and heavily relies on ACID (atomicity, consistency, isolation, durability) semantics to maintain filesystem consistency and integrity. This paper ofers an overview of Chimera's current capabilities and design. Additionally, it emphasizes the necessity for future enhancements to ensure scalability and meet the evolving demands of scientific research.

Keywords

• eol>Distributed Storage System
• Filesystem Metadata
• High-Performance Data Access
• Scalability

1. Introduction

dCache provides a system for storing and retrieving The ever-increasing amount of data that is produced huge amounts of data, distributed among a large number by modern scientific facilities like EuXFEL, PETRA III of heterogeneous server nodes, under a single virtual or LHC puts high pressure on the data management filesystem tree with a variety of standard access methods. infrastructure at the laboratories. The challenges that It strictly separates the file namespace of its data reposihave to be addressed span over the full data life-cycle: tory from the actual physical location of the datasets. The from ingest and eficient data analysis, up to long-term ifle names, attributes, and filesystem tree are managed preservation. Even though object stores, like Amazon in an internal database and exposed through a namesS3[ 1 ], become more popular, typically data is stored in pace component. By splitting a file’s metadata and data, POSIX-compliant[ 2 ] filesystems, e.g. stored as a large dCache uses a unique identifier for each file which is number of files organized in a hierarchical directory independent of the file’s name and location. By using structure. To achieve the desired performance, durability, this level of indirection, dCache can store multiple copies and parallelism, such filesystems are implemented as of a file, dynamically add or remove new locations and distributed storage clusters, that almost linearly scale the make use of external storage like S3 or tape. The system capacity and the aggregated bandwidth with the number horizontally scales with the number of data servers. By of installed data servers in the systems. One such storage adding new nodes into the system both storage capacity system is dCache[ 3 ], which is developed by Deutsches and aggregated data bandwidth grow. Elektronen-Synchrotron (DESY) in collaboration with Fermi National Accelerator Laboratory and Nordic eInfrastructure Collaboration (NeIC). 1The name comes from Pretty Normal File System[ 5 ], the original backend for file metadata in dCache 2The name comes from an animal from ancient Greek mythology with a lion’s head and fore parts, a goat’s body, a dragon’s rear, and a tail in the form of a snake.

2. Chimera Design

The namespace component of dCache is designed to address the following requirements: Unique filesystem object IDs independent from name File names are not persistent, while data is. Users can rename files, but still be able to access the original data; Metadata associated with files and directories An arbitrary metadata can be associated with files, in particular, storage system-specific information like tape name, ofset and so on; Name-to-ID and vice versa mapping By referencing files in the storage system by ID we need a possibility to find the file ID while users will operate by file names; Filesystem consistency In a highly concurrent distributed system all clients should see a consistent view of the namespace; Ability to bypass POSIX interface Though endusers typically use the POSIX interface to access the data, the system itself, as well as system administrators need an alternative path for maintenance operations or functionality beyond the POSIX standard.

The database schema contains several tables that have foreign key constrain to a main table with a list of all iflesystem objects, known as inodes (Listing 1). Note, that filesystem objects are represented by two unique IDs. One is the database auto-generated ID, which is used for reference from other tables, and one externally provided UUID-based identity, which is used by the rest of the systems to identify files on data servers. Such internal+external combination reduces the storage requirements of the database engine (int vs. char(36)) and allows the merging of multiple instances into a single one, if needed. The filesystem hierarchical view, the file system tree, is implemented as adjacency list, see Listing 2, which is well suited for single path element lookups, havely used by UNIX virtual filesystem layer[ 11 ]. An example of file the creation SQL procedure is demonstrated in Listing 3.

CREATE TABLE t _ i n o d e s ( i n o s e r i a l PRIMARY KEY , −− o b j i d i d char ( 3 6 ) UNIQUE , −− UUID u s e d

by s t o r a g e s y s t e m s t y p e i n t e g e r NOT NULL , −− o b j e c t

t y p e mode i n t e g e r NOT NULL , −− POSIX

p e r m i s s i o n s n l i n k i n t e g e r NOT NULL , −−

number o f r e f e r e n c e s u i d i n t e g e r NOT NULL , −− o w n e r

n u m e r i c i d g i d i n t e g e r NOT NULL , −− o w n e r

g r o u p n u m e r i c i d s i z e b i g i n t NOT NULL , −− o b j e c t

s i z e i n b y t e s c r t i m e timestamp NOT NULL , −−

o b j e c t c r e a t i o n t i m e c t i m e timestamp NOT NULL , −− o b j e c t ’ s l a s t a t t r i b u t e c h a n g e t i m e a t i m e timestamp NOT NULL , −−

o b j e c t l a s t a c c e s s t i m e mtime timestamp NOT NULL , −− o b j e c t l a s t m o d i f i c a t i o n t i m e ) ; ) ;

Listing 1: Inodes table

CREATE TABLE t _ d i r s ( p a r e n t b i g i n t NOT NULL , −− p a r e n t o b j e c t i d c h i l d b i g i n t NOT NULL , −− c h i l d o b j e c t i d name varchar ( 2 5 5 ) NOT NULL , −− c h i l d ’ s name i n p a r e n t d i r e c t o r y FOREIGN KEY ( p a r e n t )

REFERENCES t _ i n o d e s ( i n o ) , FOREIGN KEY ( c h i l d )

REFERENCES t _ i n o d e s ( i n o ) ,

PRIMARY KEY ( p a r e n t , name )

Listing 2: Directory hierarchy table

Other tables are used to represent extended file attributes, checksums, user-defined tags, pointers to the physical location of the file, data placement policies and other information used by dCache.

BEGIN

−− c r e a t e a new i n o d e r e c o r d INSERT INTO t _ i n o d e s ( i n o , . . . ) VALUES ( i n o , . . . ) −− c r e a t e an e n t r y w i t h a g i v e n name i n t h e p a r e n t d i r e c t o r y INSERT INTO t _ d i r s ( p a r e n t , c h i l d , name ) VALUES ( p a r e n t _ i n o , i n o , ’ obj_name ’ )

COMMIT

Listing 3: Create entry example

If the parent directory does not exist or already contains an object with the given name, then the database transaction will fail to guarantee the filesystem consistency. With such a directory structure renaming a file or directory performed by an update of a single record, and moving the whole directory subtree into a new location is independent of the number of filesystem objects and the depth of that subtree.

The database schema of Chimera allows very eficient so-called singleton queries, which are queries that return a single row, for example, when querying file attributes or checking an object’s existence in a directory, as well as a directory listing. Moreover, by exposing filesystem internals via DB query interface, some filesystem maintenance operations can be performed bypassing the POSIX

The popular NoSQL solution doesn’t have the limitations of traditional RDMS but has weaker consistency guarantees.

There are several attempts to make scalable and consistent ACID-compliant databases, so-called NewSQL databases, for example CockroachDB[ 13 ]. By supporting the PostgreSQL wire protocol CockroachDB is almost a drop-in replacement for dCache’s namespace database.

Unfortunately, basic performance tests have shown the poor performance of CockroachDB in comparison to a stand-alone PostgreSQL server. 4. Future work interface, thereby not being limited by it. An example of such operations is the calculation of storage usage per user, finding directories with an abnormally large number of child objects (files or directories), or data deduplication based on checksums stored in one of the auxiliary tables. Nevertheless, there are downsides to such a design as well. Though Chimera is very efective for lookups in a single directory, accessing the files by full path requires multiple sequential lookups per file path element.

Such accesses are typical for URL-based protocols, like HTTP. Though nested sets based representation of the iflesystem hierarchy is more eficient for fullpath-based access, it comes with a high performance penalty when a new tree nodes, e.g. new directories, are created. For that reason, even though the majority of remote data transfers are performed by the HTTP protocol, the constantly changing large directory trees and the dominance of POSIX-like access from the local compute clusters (the direct file system mounts through NFSv4.1[ 12 ]) define the optimization priorities.

Though the system’s scalability satisfies today’s requirements, it might become a bottleneck for upcoming detector and accelerator upgrades, which are expected by PETRA-IV in 2028, or the High Luminosity phase of the LHC starting in 2029. According to the ATLAS Software and Computing HL-LHC Roadmap[ 14 ], in the High-Lumi LHC mode, the ATLAS experiment will record 7-10 times 3. Transactional requirements more data than today, thus putting higher demand on the storage system, including the metadata component. The As mentioned above, Chimera highly relies on ACID capa- same data rate growth is expected by other experiments bility of the underlying database. Moreover, some filesys- at LHC as well. To handle the amount of generated events, tem operations operate on multiple objects (records) at it is common practice for experiments to write data in the same time and therefore require atomic operations multiple parallel streams, often into a single directory. on multiple records in a single transaction. For exam- For the metadata service (as well as for storage and ple, moving a file from one directory to another should database systems), there are two important metrics: ladelete the entry in the source directory, if exists, delete tency and throughput, which measure diferent aspects the file with the same name in the destination directory of system performance. Latency refers to the time it (this already includes updates on a directory structure takes for a single request to be completed. Low latency and list of all inodes), and, finally, create a new record in is desirable because it means operations are completed the destination directory. To ensure consistency, all four quickly, resulting in faster response times for applicaupdates must be performed as an atomic operation and, tions. Throughput, on the other hand, measures the rate thus should be executed as a single transaction. How- at which multiple requests are executed simultaneously. ever, strong consistency guarantees provided by rela- Though ideally, we want to improve both, throughput tional databases introduce scalability limitations: plays a major role, as it directly afects the system’s scalability. • Database transaction serialization in workloads, With the current design, a large number of concurrent where a large number of concurrent clients up- writes end up in database update serialization, as demondate a single directory. strated with a synthetic load test in Figure 3, where all • Huge tables require large disk storage, memory processes are waiting for an exclusive lock on the same and additional CPU resources on database servers. database entry, resulting in low overall throughput, even In HA clusters, all nodes must fulfil such require- if the rest of the system scales horizontally or vertically, ments. i.e. scales with a number of nodes in the system or more • Clients that are querying diferent sets of meta- powerful machines, respectively.

data can’t be dynamically served by multiple servers after the connection to a database has been established, thus dynamic workload distribution is not possible, which results in overloaded servers on one side, and underutilized servers on the other.

The POSIX I/O interface was defined in the late 80’s

for single-node systems with a single disk and, thus, was not designed for the highly concurrent computing environments we have today. To reduce the load on the metadata server some distributed storage systems don’t follow POSIX semantics and provide a special application-level library that provides access to the stored data. Google’s File System (GFS) [ 15 ] doesn’t have a concept of a parent directory and stores the files by full name, thus file creation does not require a write lock on a parent directory3. Such systems might provide a high update rate of file metadata, however, they require application modification and can not be used as a general-purpose storage solution in multi-science environments.

The dCache’s namespace is not the only attempt to implement the filesystem’s metadata part in a relational database. The TableFS[ 16 ] follows a similar approach, though it uses a single table for the list of all inodes and the directory tree structure. However, by providing a strong consistency through back-end DB implementation the concurrent updates in the single directory TableFS are serialized as well.

To address those data management challenges, the dCache developers are investigating metadata catalogue scalability requirements to propose a solution that will have the potential to replace or significantly improve the existing namespace component of dCache, in particular: • Estimate the typical workload on the metadata services and identify the scalability limitations of the existing solution. • Identify workflows that require strict POSIX and near-POSIX compliance. • Propose a design of a new metadata catalogue, that is at least a factor of 10 more scalable in the number of filesystem objects and overall throughput, i.e. operation per second, without compromising the filesystem integrity. 3Colossus, the successor to the Google File System might have a diferent architecture. The information about Colossus’s design is publicly not available

Tough POSIX compliance is typically expected by the storage systems, in many cases, it is not required by all the applications. Even though there are some studies to understand the requirements of the scientific application, they are typically focused on the I/O path and pay very little attention to metadata-related operations[ 17 ]. Thus, a detailed analysis of scientific applications might provide opportunities to soften the POSIX compliance requirements and improve metadata scalability by implementing eventual consistency for some operations without compromising the filesystem integrity.

The R&D activity by the dCache team aims to utilize existing database technologies and widely available techniques to implement a new scalable metadata service for scientific data repositories.

References

[1] Amazon , Amazon S3 , 2023 . URL: https://docs.aws. amazon.com/s3/index.html.

[2] Ieee standard for information technology-portable operating system interface (posix(tm)) base specifications, issue 7 , IEEE Std 1003 . 1 -2017 (Revision of IEEE Std 1003 . 1 - 2008 ) ( 2018 ) 1 - 3951 . doi: 10 .1109/ IEEESTD. 2018 . 8277153 .

[3] Mkrtchyan , Tigran, Chitrapu, Krishnaveni, Garonne, Vincent, Litvintsev, Dmitry, Meyer, Svenja, Millar, Paul, Morschel, Lea, Rossi, Albert, Sahakyan, Marina, dcache: Inter-disciplinary storage system , EPJ Web Conf . 251 ( 2021 ) 02010 . URL: https://doi.org/10.1051/epjconf/202125102010. doi: 10 .1051/epjconf/202125102010.

[4] A. S. Foundation , Apache zookeeper, 2023 . URL: https://zookeeper.apache.org.

[5] P. Fuhrmann , A perfectly normal namespace for the desy open storage manager , 1997 . URL: http: //www-zeuthen.desy.de/CHEP97/paper/409.ps.

[6] M. Gasthuber , T. Mkrtchyan , P. Fuhrmann , Chimera - a new, fast, extensible and Grid enabled namespace service , in: 14th International Conference on Computing in High-Energy and Nuclear Physics , 2005 , pp. 1180 - 1182 . doi: 10 .5170/ CERN-2005- 002 . 1180 .

[7] PostreSQL, PostreSQL, 2023 . URL: https://www. postgresql.org/.

[8] T. H. D. Group , HSQLDB, 2023 . URL: https://hsqldb. org/.

[9] M. Fowler , Strangler pattern, 2004 . URL: https: //martinfowler.com/bliki/StranglerFigApplication. html.

[10] T. Haerder , A. Reuter , Principles of transactionoriented database recovery , ACM Comput. Surv . 15 ( 1983 ) 287 - 317 . URL: https://doi.org/10.1145/289. 291. doi: 10 .1145/289.291.

[11] S. R. Kleiman , Vnodes: An architecture for multiple file system types in sun unix , in: USENIX Summer , 1986 . URL: https://api.semanticscholar. org/CorpusID:30546296.

[12] D. Noveck , M. Eisler , S. Shepler , Network File System (NFS) Version 4 Minor Version 1 Protocol, Number 5661 in Request for Comments , RFC Editor, 2010 . URL: https://www.rfc-editor. org/info/rfc5661. doi:10 .17487/RFC5661.

[13] CockroachLabs, CockroachDB, 2022 . URL: https: //github.com/cockroachdb/cockroach.

[14] A. Collaboration , ATLAS Software and Computing HL-LHC Roadmap , Technical Report , CERN, Geneva, 2022 . URL: https://cds.cern.ch/record/ 2802918.

[15] S. Ghemawat , H. Gobiof , S.-T. Leung, The google ifle system , in: Proceedings of the 19th ACM Symposium on Operating Systems Principles , Bolton Landing, NY, 2003 , pp. 20 - 43 .

[16] K. Ren , G. Gibson, TABLEFS: Enhancing metadata Eficiency in the local file system , in: 2013 USENIX Annual Technical Conference (USENIX ATC 13) , USENIX Association, San Jose, CA, 2013 , pp. 145 - 156 . URL: https://www.usenix.org/conference/ atc13/technical-sessions/presentation/ren.

[17] C. Wang , K. Mohror , M. Snir , File system semantics requirements of hpc applications , in: Proceedings of the 30th International Symposium on HighPerformance Parallel and Distributed Computing , HPDC '21, Association for Computing Machinery, New York, NY, USA, 2021 , p. 19 - 30 . URL: https: //doi.org/10.1145/3431379.3460637. doi: 10 .1145/ 3431379.3460637.