German-Russian Astroparticle Data Life Cycle Initiative is an international project whose aim is to develop a distributed data storage system that aggregates data from the storage systems of di erent astroparticle experiments. The prototype of such a system, which is called the Astroparticle Physics Distributed Data Storage (APPDS), has been being developed. In this paper, the Data Aggregation Service, one of the core services of APDDS, is presented. The Data Aggregation Service connects all distributed services of APPDS together to nd the necessary data and deliver them to users on demand.
The amount of data being generated by various astroparticle experiments such
as TAIGA [
Such a prototype has been being developed and is called Astroparticle Physics Distributed Date Storage (APPDS). The design of the system is based on two key principles. The rst one is creating no interference with existing data storage systems of the physical experiments. This principle is critical since most existing collaborations in astroparticle physics, not only the collaborations of TAIGA and KASCADE-Grande experiments, have been historically using their own stack of technologies and approach to data storage. Any change to the existing data storage systems might cause potential problems which are hard to be discovered. The second principle is using no computing resources at the sites of the storage systems to handle users' queries. This principle leads to creating a global metadata database where the data description from all storage systems is aggregated in one place. All searching and ltering operations are performed within the global metadata database. The operation results are delivered to users via a web interface. Actual data transfer from the existing data storage systems takes place only when users want to access the inside content of a le. Thus, APPDS causes no load to the computing resources at each site; the only load on the data storage systems is data delivery.
This article is organized as follows. In the second section, the architecture overview of APPDS is presented. Section 3 is dedicated to the design of the Data Aggregation Service, which is the core of APPDS, and the stack of technologies that have been used. In conclusion, the current state of the service is presented. 2
The architecture overview of APDDS is presented in Fig.1. S1, S2, S3 are data storage instances of the physical experiments. In1 is the original data input. In2 indicates the case when the original data from In1, which are already stored in a storage instance, are being reprocessed. At the level of data storage instances, to preserve the original data processing pipeline, a program called Extractor (E1) is injected into the pipeline. In most cases, input data are les. After standard processing, the les are passed to the Extractor. The Extractor retrieves metadata from the les using the metadata description (MDD) provided by the development groups of the physical experiments, sends the metadata to the Metadata Database using its API, and passes the les back to the pipeline. If the data need to be reprocessed, the same pipeline is applied but with a di erent type of the Extractor (E2).
The les of each storage instance are delivered to the Data Aggregation
Service by the Adapter, which is a wrapper of the CernVM-FS server [
To retrieve necessary les, the user forms a query using the web interface provided by the Data Aggregation Service. When the Data Aggregation Service receives the user query, it asks the Metadata Database for the answer. When the Metadata Database answers, the Data Aggregation Service generate a corresponding response and delivers it to the user.
All components of APPDS are talking to each other via RESTful API [
There are two types of queries that a user can make using the Web Interface of Data Aggregation Service. Identi cation queries are used to retrieve the basic information about the resources provided by APPDS such as facilities, clusters, detectors, data channels, data authorities, permissions, and available data.
(CernVM-FS
ech iss .2aC it/H
M
3.LookGupra/ApnhsQwLer Search queries are used when the user wants to look for the available data using a list of lters like data availability interval, energy range, facility location, detector types, data channel speci cation, weather condition, etc. Typical data lookup scenarios that users might create are: { data obtained by one facility or all available facilities for a certain period; { season data which start from September to the end of May; { data obtained in a testing period or a speci c run; { regular monthly or weekly data.
The obvious choice to implement the dialogue between the Web Interface and
the Aggregation Service as separate microservices is to create a RESTful API.
While identi cation queries are the best candidates to be implemented using
a set of endpoints as in the standard RESTful API, the search queries with
their complex lters are not. Using the standard RESTful API to implement
the search queries leads to complex query sets containing redundant data that
users do not need. The best approach to implement search queries is to use the
Facebook GraphQL [
Query Processing Pipeline
Whenever the Core Controller receives a query from the GraphQL Backend,
it calculates the query checksum using the MD5 algorithm. The checksum is
used as the query ID. After that, the Core Controller check against the Cache
Manager to nd if such a query is already registered. If not the Core Controller
registers the query with the Cache Manager. The Cache Manager is implemented
based on Redis [
If the response to the query is already cached and the active period does not expire, it will be delivered to the user. If the response to the query has not been cached, the Core Controller forwards the query to the Metadata Database. The Metadata Database generates a SQL-query according to the original query and executes it. Depending on the data sources, the response from the Metadata Database can vary. The response, including data from the TAIGA experiment, is a set of les which match the lters speci ed in the original query. The Metadata Database also calculates the checksum of the response, adds it to the nal response and sends it to the Core Controller. The Core Controller, in turn, appends the response to the query in the Cache Manager. After that, the Core Controllers sends the response to the Web Interface so that the user could look at it quickly.
At the same time, the Core Controller starts preparing the full response for delivery. Files from all Data Storage instances are exported to the Data Aggregation Service as separate CernVM-FS repositories. All repositories are located inside a Global Mount Point where each subdirectory is the mount point of each repository. To prepare the data, the Core Controller nds the les in the Global Mount Point and copies them to a directory in the Local File Bu er. The directory is then con gured as a CernVM-FS repository for export. The name of the repository is the same as the query ID. If the user query requires not the original les but a composition of them, the Core Controller scans through the original les, composes a new set of les, and puts them into the directory. Later, if the user wants to work with the les locally, she can mount the prepared CernVM-FS repository to her computer. The Core Controller also generates an archive containing all les. The archive is put into the repository and can be downloaded separately. If the full response to a query is not used more than a speci c time, it will be deleted from the Local File Bu er and the Cache Manager. 4
Currently, a beta version of APPDS, including the Data Aggregation service, the Metadata Database, the Adapter, and the Web Interface, has been implemented and being tested against the storage system of the TAIGA experiment. In the next release, support for the storage system of the KASCADE-Grande experiment will be included. The rst working prototype of the system is planned to be released this fall. The proof-of-concept tests and performance benchmarks are also planned after the release.