The big data revolution has overturned well-established approaches to data analysis and intensi ed the demand for access to heterogeneous data. Modern developed methods allow for the extraction of new knowledge from the data, which enables researchers to approach many previously unsolved mysteries. This trend, observed in many areas of human activity, is also tangible in the eld of astroparticle physics. Combined analysis of various experimental data allows researchers to derive deeper insights into the processes occurring in the universe and extend the borders of our knowledge about nature. Providing the infrastructure for such investigations is a topical issue of the astroparticle physics community. In this report we examine a service for the aggregated retrieval of heterogeneous data from distributed storages of numerous astroparticle physics experiments. We describe its architecture, available data, principles of functioning and interaction with users and data centers.
Since the moment of their discovery in 1912, cosmic rays (CR) have been studied quite thoroughly. Nevertheless, many mysteries related with them remain the subject of active community research, including CRs spectrum and mass composition, behaviour of matter at ultra-high energies not achievable in terrestrial accelerators, mechanisms of CRs acceleration and propagation as well as their origin. Numerous experiments of astroparticle physics around the world record the particle cascades generated in the interactions of relativistic cosmic rays with the atmosphere of planet Earth.
A valuable trend in the search for insights into the above-mentioned matters, which includes development of methods for combined analysis of observations from multiple components (messengers) of cosmic radiation, is called MultiMessenger Astroparticle Physics.
The other modalities for combined analysis are encouraging as well, as they allow an increase of the statistical data of observables and positively in uence the accuracy of the analysis.
Thus, development of a united infrastructure for aggregated access to
heterogeneous experimental data becomes of much importance. Such an
infrastructure would integrate data from heterogeneous geographically distributed
storages, be resilient, horizontally scalable, and support the entire data life cycle
from creation to archiving or destruction, i.e., would satisfy the FAIR (
ndableaccessible-interoperable-reusable) [
In the framework of the international German-Russian Data Life Cycle
initiative [
Data aggregation framework
The developed data aggregation system includes such elements as the setup (facility), data center of the experiment (storage), data extractor, adaptor, application programming interface by GRADLC (GRADLC API) and metadata database (MDDB). In a simpli ed form, the interaction of these elements is shown in Fig. 1.
Let us examine the elements from Fig. 1 and their interaction in more detail. A storage is a data center or a database of an experiment in astroparticle physics, where data of any reconstruction level registered by experimental setups (facilities) are recorded.
Infrastructures for data storage can be loaded with internal search queries of collaborators, processing new data, and with other computationally intensive tasks. Under these conditions, performing complex search queries on the storage side is undesirable, since it can lead to a decrease in the performance of the storage. To reduce the search payload, an approach was proposed to use a simpli ed database on the side of the aggregator, called metadata database (MDDB), which stores metadata for the data from experimental storages.
Facility <...> Facility
Data retrival by uuid
Returns a download link Storage Adaptor
Extracts metadata
Extractor Requests data with particualr uuids in particular format
Returns uuids
MDDB GRADLC API
Requests uuid by metadata request data file
User
This approach is based on a two-level metadata system described in [
A program called extractor is used to retrieve information from the repositories to MDDB. The schedule of running the extractor depends on the frequency of data updates in the repository.
When a request is received to retrieve data from the system, a search for the relevant parameters is performed in MDDB in order to nd universally unique identi ers (UUID) that correspond to the criteria of the events. Next, a list of unique identi ers for each experiment is passed to the aggregator, which connects to the repository, extracts the necessary data and transfers it to the aggregator.
The interaction of the user with the system and with individual components within the system is organized through the API, which will be discussed in detail in section 2.4.
It is important to note that in order to maintain simplicity, Fig. 1 shows the interaction in the case of a single data storage. However, the approach we developed allows interaction with multiple repositories, and it is easy to add new ones. In this case, a custom extractor and adapter correspond to each new data storage.
Currently, the users of the service have access to the data of Tunka-133
experiment, KCDC, and TrVo. Users of the APPDS [
Currently, the system provides access to the data from the following
experiments: KASCADE, KASCADE-GRANDE [8], LOPES [9], Tunka-133 [
A system of joint data retrieval allows the user to upload data from several setups in a overlapping range of values. The list of available cuts for all the data available through the site is presented in table 1. The available range of values can be seen in the GRADLC API o cial documentation [13].
A list of possible selection parameters for simulations is presented in table 2. To analyse the presented data, one can use the analysis framework [14], integrated into KCDC also within the GRADLC project. A more detailed description of the data can be found on the o cial websites of the experiments and related data centers or virtual observatories, as well as in the o cial articles of the mentioned collaborations. 2.3
Asynchronous data processing and interaction with storages Let us examine in more detail the behaviour of the system when it receives a data retrieval request. In Fig. 2 the user sends a request to the aggregator using the GRADLC API. On the aggregator side, a unique request identi er is generated, that is later used by the system to perform all actions associated with its processing. After that, the request is also added to the aggregator database with the request status of "Scheduled", and the request identi er is returned to the user to allow further actions associated with it (see section 2.4).
On the backend, the scheduler daemon notices that the new query has been added to the aggregator database, and starts its processing. For this purpose, if there are free computing resources on the server, a process is created to retrieve the requested data from the storages, described in more detail in Fig. 3. The status of the request is changed to "Running". It should be noted that the length of the queue and the number of simultaneously launched tasks are limited both on the side of the aggregator and on the side of the storages for technical reasons, and this may constitute a bottleneck of the system.
User Data retrieval
request Get dloinwknload
GRADLC aggregator
UUID generation dCowrleinnalkotea d Di"AnBSdtocwdhwiretheedqbusutlAeaedtPsu"tIs
Web API DB monitoring [new request is found]
no s e y Create new retrieval process
Thread 1 ./DataAggr
Scheduler
Process pool monitoring no stCathuasntgoe"Fprinoicsehsesd" or "Failed" yes [process is finished] Processes pool ... Thread N Thread N+1 ./DataAggr ./DataAggr [finished successfully?]
no yes Save datafile Remove incomplete datafile
The scheduler monitors running processes and upon completion of the process changes its request status in the system to "Failed" in case of the failure or to "Finished" in case of successful completion. For successfully completed processes, the query results are written to the server and archived for further download by the user. For the unsuccessful processes, intermediate les created during the processing are removed.
Let's take a closer look at the process of data extraction by user request, shown in Fig. 3.
The scheduler passes the query parameters to the instance of the Data aggregator process that executes the query to MDDB and receives the UUID list corresponding to the speci ed parameters as its response. Next, the aggregator request the adapters of the necessary data storages for direct upload the necessary experimental data.
It should be noted that new repositories can be included quite easily: this requires a relatively easy-to-write repository adapter, a metadata extractor 2.1 and a repository data indexation with UUID.
run return data Scheduler
request by uuids in neсessary DBs
Mongo DB request with parameters return uuid Tunka-133 Adapter
Application programming interface In order to provide user interaction with the aggregator an application programming interface GRADLC API was implemented, employing JSON-RPC 2.0 [15] remote procedure call protocol with data transmission over http.
There are ve methods one can request through the aggregator API: data extraction, request status, list of requests, cancelling request, and data download. One can nd examples of the request and detailed explanations in the o cial documentation [13].
Two days after the request was made related data is deleted from the server automatically and the query gets status "Expired". There are six possible statuses a query can have in the system: "Scheduled", "Running", "Finished", "Failed", "Expired", and "Deleted".
Web user interface For achieving a better user experience an extended web graphical user interface (Web GUI) was developed. Some examples of user interaction with it are shown in Fig. 4-5. List of data-upload tasks The developed software supports aggregation of heterogeneous data from a variety of geographically spread data storages mentioned above.
The valuable features of the service, such as faster data retrieval employing a metadata data search concept, multiple lters for data search, asynchronous multithread data processing as well as di erent possibilities of interaction between the system and the user make it a valuable product, which could be used as a service for automatic data retrieval in large scale research projects as well as a stand-alone application for individual outreach projects.
Our future plans include employing advanced data management tools for message broking and system status monitoring, as well as possible integration of other physical experiments into the GRADLC data aggregation platform.
More information about the project can be found at the project page [16]. 7. Budnev, N. et al. - TAIGA Collaboration: TAIGA the Tunka Advanced Instrument for cosmic ray physics and Gamma Astronomy - present status and perspectives.
JINST 9 (2014), C09021 8. Homepage of KASCADE-Grande, https://web.ikp.kit.edu/KASCADE/. Last accessed 27 June 2020 9. LOPES | A LOFAR Prototype Station Homepage, https://www.astro.ru.nl/lopes/. Last accessed 27 June 2020 10. Schroder, F. G. et al.: Tunka-Rex: Status, Plans, and Recent Results. In: EPJ Web of Conferences 135, 01003 (2017) 11. KCDC Announcements - ChangeLogs, https://kcdc.ikp.kit.edu/announcements/changeLogs/.
Last accessed 27 June 2020 12. Schoo, S.: Energy Spectrum and Mass Composition of Cosmic Rays and How to
Publish Air-Shower Data. PhD thesis. KIT, Karlsruhe (2016) 13. GRADLC API Documentation, http://141.52.67.147:55000/web/doc/. Last accessed 27 June 2020 14. Polgart, F.: Analysis framework for KCDC. In: this proceedings, pp. ?{?. Publisher,
Location (2020) 15. JSON-RPC 2.0 Speci cation, https://www.jsonrpc.org/speci cation. Last accessed 27 June 2020 16. GRADLC project web page, http://141.52.67.147:55000. Last accessed 27 June 2020