AstroDS is a distributed storage for Cosmic Ray Astrophysics. The primary goal of Astro DS is to gather data measured by the instruments of various physical experiments such as TAIGA, TUNKA, KASCADE into global storage and provide the users with a standardized user-friendly interface to search for the datasets that match certain conditions. AstroDS consists of a set of distributed microservices components that communicate with each other through the Internet via REST API. The core component of AstroDS is the Data Aggregation Service that orchestrates other components to provide access to data. The development process of AstroDS started in 2019. This paper describes the current design and implementation of the Data Aggregation Service and also the bene ts it brings to the astrophysical community in the early state.
Russian{German Astroparticle Data Life Cycle Initiative [
Copyright ' 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
AstroDS consists of a set of distributed microservices that interacts with
each other to provide a smooth experience for users in data acquisition. One of
the core services of AstroDS is the Data Aggregation Service (the Service) that
coordinates others to ful ll data queries from users [
Input Query Web UI
HTTP Server
Cache Manager
Core Controller Repository Manager
Metadata Catalog Storage
Instances
Data Aggregation Service
The overall view of the Service's components is presented in Fig. 1. On the frontend side, through the Web User Interface, the Service receives data lookup queries from users. The content of a query contains a set of search lters de ned by a user. For example, the query can contain IACT01, which is the code name of a facility of the TAIGA experiment, two timestamps identifying the start and the end of a certain period, the weather condition when the data were collected at the experiment's site.
On the backend side, the Web User Interface is served by an HTTP Server. The HTTP Server serves as a proxy layer that passes the queries to di erent instances of the Service. Within each instance of the Service, the Core Controller is responsible for processing queries. Upon receiving a query, the Core Controller always checks with the Cache Manager if there is a response already generated for the query. To uniquely identify a query, the Core Controller uses the MD5 algorithm to calculate the hash of the query using its content. A response to a query is a list of les containing data that matches the lters de ned in the query. If the response to a query is not cached, the Core Controller will forward the query to the Metadata Catalog that stores the metadata of all available datasets. The Metadata Catalog, in turn, makes a search query against its database and returns a response to the Core Controller. After receiving the response from the Metadata Catalog, the Core Controller passes it to the Cache Manager for caching and requests the Repository Manager to generate the les in the response using data from the Storage Instances each of which is mounted to the machine where the Service is running. File generation is carried out asynchronously. 3
The Web UI was implemented as a single-page application based on React [
Since the history of all queries made by the user is stored in the local storage of the browser, it is possible to review the response to a query. While reviewing the responses, the user can make a list of les from di erent queries just by selecting them. The selected les later can be downloaded as a single archive. An illustration of the Web UI is shown in Fig. 2
The Core Controller was implemented as a microservice based on the Django
framework [
All queries are cached by the Cache Manager to reduce the response time. The cache was implemented as a big hash table where the key is the MD5 hash of a query, and the value contains the response to the query and other metadata such as creation time, last used timestamp. Queries are discarded from the cache by the Least Frequently Used (LFU) policy: the Cache Manager counts how often a query is used; those that are used least often are discarded rst. Currently, the expiration time of a query is seven days.
The Repository Manager was implemented as a separate microservice with its REST API. After receiving a le generation request from the Core Controller, the Repository Manager creates an empty CernVM-FS repository, copies the original les from the storage instances to the newly created repository, and publishes it. When a repository is ready, the Repository Manager sends a short text message to the Core Controller via the WebSocket channel.
To implement the communication via WebSocket, the RabbitMQ Message
Broker [
Facilities of the physical experiments mentioned above (TAIGA, KASCADEGrande) were successfully integrated into the AstroDS system. Using the Web UI of the Data Aggregation Service, users can already make a complex query that combines data from di erent facilities of the experiments. The KASCADEGrande experiment has its own service similar to the Repository Manager, so components of the Data Aggregation Service work with that service to generate les. There are ongoing works to make the Data Aggregation Service stable in production use. In the future, there are plans to integrate more physical experiments into the AstroDS platforms and implement more lters that give users a exible way to search for data. There are also plans to conduct outreach activities to increase the number of users.