Large-scale experiments in astroparticle physics are usually operated several decades by international collaborations of partly several hundreds of scientists. Experiments launched some decades ago, trying to make their data publicly available, su er from the fact that their data structures cannot be evaluated using modern information technologies. To overcome this situation and to guarantee a FAIR ( ndableaccessible-interoperable-reusable) [2] data preservation, the data must be restructured and reformatted. A step in this direction is to provide the data and meta-data as well as the tools to analyse the measured data of the meanwhile dismantled cosmic ray experiment KASCADE, which operated from 1996 to 2013. The project to make the entire scienti c data public is called the 'KASCADE Cosmic Ray Data Centre' (KCDC, https://kcdc.ikp.kit.edu). The activities within KCDC are used as blueprint for a sustainable data life cycle including aspects of data curation in astroparticle physics. With this paper we give an overview of the current status of the KASCADE Open Data Publication via the KCDC web portal with focus on the adaption of the initial structure of the KASCADE data for the KCDC data publication.
A major topic of Astroparticle Physics is to investigate the nature of the
cosmic radiation manifesting in many ways. The detection of high-energy cosmic
rays is performed by the registration of extensive air showers, i.e. the secondary
? Supported by KRAD, the Karlsruhe-Russian Astroparticle Data Life Cycle Initiative
(Helmholtz Society Grant HRSF-0027). The authors acknowledge the cooperation
with the Russian colleagues (A. Kryukov et al.) in the GRADLC project (RSF
Grant No. 18-41-06003) as well as the KASCADE-Grande collaboration for their
continuous support of the KCDC project.
particles generated by the primary cosmic ray when entering the atmosphere. In
various air-shower experiments located all over the world the cascades of
particles are detected, generated in interaction processes of the relativistic cosmic
rays (mainly fully ionized atomic nuclei, but also cosmic gamma-rays and
neutrinos)with the molecules of the Earth's atmosphere. As these air-showers consist of
a huge number of particles which are spread over a vast area, large detectors are
required to measure certain characteristic parameters. Dedicated reconstruction
algorithms [
The combined analysis of observations of various components of the cosmic
radiation like charged particles, gamma rays and neutrinos is widely known as
'Multi-Messenger Astroparticle Physics'. From this, currently as hot topic
classi ed research eld, we hope to gain new and exciting information to extend our
knowledge of the origin and transport of what we understand as cosmic
radiation. The Multi-Messenger Astroparticle Physics requires access to data (in a
reasonable and standardized format) of the diverse experimental installations.
Our studies described here are important steps towards establishing a global
data and analysis centre [
From Proprietary Experimental Data to an Easy
Accessible Open Data Format
KASCADE-Grande [
The KASCADE-Grande experiment consisted of four major detector
components. An array of 252 detector stations housing separate electron and muon
detectors, a central detector for measuring the hadron component with
additional muon detection area, a tunnel with streamer tubes to record individual
muon tracks and, a second array with 37 detector stations for electron detection
which extends the e ective KASCADE array area from 200 200 m2 by a factor
of 10 built in 2003. Furthermore, an array of 30 radio antennas is co-located with
the KASCADE array (LOPES [
After the shutdown of the KASCADE experiment the number of scientists who can maintain the databases with old software on old computers degraded fast. The only chance to get the results comprehensible for other experiments is to start a data curation process and feed the data to a system where they can be kept maintained. 3
KASCADE Data Acquisition and Data Structure
The KASCADE Data Acquisition System was designed and set up in the early
nineties of the last century using the most modern types of databases and
software available. By that time all o -line programming in high-energy physics was
carried out using Fortran 77 programming language. Although this language
offered some advantages compared to other common languages at that time, it
su ered from a lack of dynamic data structuring facilities. To overcome these
disadvantages CERN ZEBRA [
Below a le header holding information about the current run, properties like start time, detector status, calibration settings etc., the event blocks are organized sequentially, each consisting of an event header and the hierarchically arranged ZEBRA banks for the event data of every detector component being part of the respective event, followed by a direct access table information on number of events and the position of the rst data set. Each of the various components of the KASCADE-Grande detector system was designed as an independent experiment with its own data acquisition and control system. Besides the control and monitoring of the hardware, the preparation of the data recorded and the transmission to the Central Event Builder was part of each system. The main task of the Central Event Builder is to merge time information and the recorded data of all running detector components and to transfer it to a mass storage system. The event builder is invoked by a trigger signal from any of the various trigger sources of KASCADE. The conditions to generate a trigger are de ned in the local processes of the detector components without any control by the event builder. Data blocks with time labels within a window of 10 s are merged to one event. If a detector component does not supply data for more than 2 minutes, a hardware error is assumed. Then the corresponding component is removed from the data acquisition and a new run is started. Altogether about 1.7 billion events have been recorded during the lifetime of KASCADE from 1996 to 2013 stored in more than 50.000 raw data les, consuming 4 TB of storage space, and analysed with the KASCADE data reconstruction program KRETA. The data have been archived on a tape robot of the central computing department and copied in portions to local disks for fast access during data analysis. Meanwhile all measured and analysed data are stored locally on a Raid system and on the IBM Tivoli Storage Manager of KIT-SCC (Steinbuch Centre for Computing) for long term preservation. 4
The measured air showers in KASCADE are analysed using the reconstruction program KRETA (Kascade Reconstruction for ExTensive Airshowers), which reads the raw data, performs the calibration and reconstructs the basic shower observables, storing all the results in the form of histograms and vectors of parameters (n-tuples).
The calibration parameters and the geometry of the detector layout are stored
in a time dependent database called CERN HEPDB [
Data sets recorded with irregular hardware conditions (bad runs) have been identi ed in a preanalysis step and removed for the nal reconstruction.
Data quality is commonly described as a state of accuracy for appropriate
use of data. The re-usability of an astroparticle experiment dataset relies on a
high quality lter of the raw data. Only the researcher can take into account all
details of data acquisition and data transformation. The raw KASCADE data
are reduced by a factor of four by applying quality cuts within the data analysis
procedure, mostly because low energetic events with a too small signal cannot
be reconstructed properly. All these cuts were applied during the calibration
phase on an event-by-event basis as well as the energy deposits corrections for
the detector stations agged as \not working" or \in saturation". Calibration
data, correction data and information on the detector layout are taken from a
HEPDB data base where more than 100 di erent time dependent calibration
tables are stored. The reconstruction procedure itself is then done in 3 iteration
levels. A rough estimation of shower parameters in level 1 is followed by a more
sophisticated analysis in level 2 where the results of level 1 are taken as starting
values for level 2. Level 3 aims for an even more tuned accuracy using the
results of level 2 and delivers the nal reconstruction results. These results for
each iteration level are stored in so called ntuples making use of the HBOOK
package [
An ntuple is like a table, where all the variables belonging to a certain event
are columns while each event is a row. In this form it is easy to generate one or
multi dimensional projections of any of these several hundred variables. Storing
requirements become signi cant for large event samples. The most important
output parameters stored in ntuples for KASCADE are:
{ the reconstructed position of the shower core,
{ the reconstructed shower direction,
{ the reconstructed number of electrons and muons,
{ most probable reconstructed primary energy E0 based on a certain variant
of the reconstruction process (is always based on a theoretical model of
hadronic interactions incorporated in simulations [
In the early years of KASCADE the resulting hbook les have been visualized
using the CERN PAW framework (Physics Analysis Workstation [
KCDC portal as a demonstrator for Big Data Analytics
in astroparticle physics
The main goal of KCDC [
The large amount of data stored in ROOT les makes it impossible to o er
the entire data directly for download. Thus we choose to ll the reconstructed
data and meta data into a database and install a web portal, named the KCDC
data shop [
The KCDC web portal is publicly accessible, no special software installation is required on the user side. All data are o ered for ftp download in ROOT, which is standard data format in High Energy Physics, or in HDF5, mostly used to store big amounts of data, or in ASCII for simple education examples.
Most of the parameters stored in the ROOT les gained from the KASCADE data reconstruction with KRETA can only be used by experts with detailed background knowledge of the detector geometry and the detector properties. In order to give a wider audience the opportunity to use these data, they must be prepared in such a way that they can be handled without special knowledge.
In modern terms this procedure is known as data curation. Curation includes a range of activities and processes done to create, manage, maintain, and validate data elements. Speci cally, data curation is the attempt to determine what information is worth saving and for how long. The data curation work ow is determined from data quality management, data protection, life cycle management and data movement. Therefore the routine lling KASCADE data in the MongoDB of KCDC determines which data should be published in that speci c version of KCDC. The KCDC Major Version Number is an indicator for a complete dataset. Any changes within the database will be indicated by a new version number and we can only guarantee a valid representation of KASCADE using data from the same published versions.
Another important point for this procedure is that in KRETA each detector component generates its own ROOT les because no data merging has been applied. This disadvantage we overcome by publishing the combined data analysis in a new KCDC data shop presently under construction. For the time being, we merge the data of the di erent components when lling the MongoDB.
As the KASCADE Events can be uniquely identi ed by their Run- and Event numbers, the lling routine uses these two parameters to merge the data sets from the di erent components of KASCADE and some additional information from separate tables like exclude lists for special events. Most of the space required in the MongoDB is occupied by the data arrays holding information on e/ and -energy deposits of each of the 252 KASCADE detector stations and ‚DATA‘ Collection d -i 1 t n e v E a t a d a t e m t n e v e 1 tnt A ne e n n o o p p mm oo tcc.t c . c e t e e t d e d :
: X t n e n o p m o c t. c e t e d d i2 t n e v E a t a d a t e m t n e v e 1 tnt 1 ne e n n o o p p mm o o tcct. c . tec e e t d e d ‚ARRAYS‘ Collection ]} s itednonpoACm iilitf[ttr{eessnono itednnopoXCm a t S ]} s e itr n e f o ilt s [{ s n o it a t S the arrival time of the rst particle hitting the respective station and producing a valid signal in the detector electronic. The structure scheme of the data stored in the MongoDB is shown in g. 3. In MongoDB we have two collections named `DATA` and `ARRAYS`, where the rst is lled with (meta)data from the reconstruction process with information on parameters like event number, event time etc. and shower parameters like core position, angle of incidence and number of particles separately for each detector component. In the second, the `ARRAYS`, relevant data from every detector station are kept. 6
Publishing data for other scientist enhances the sensibility for correct documentation and cross-checking the results. Using only one source of data for researchers from the KASCADE collaboration and for external researchers led automatically to a high integrity of the published data sets and increased our con dence in the data sets published. Published data sets in experimental physics underlie changes whenever analytical methods are improved or errors are discovered. A change of the version number indicates a non-semantic change of the data sets whereas the elements e.g. a speci c event, is still part of the data set. A unique UUID, implemented in the next release, represents such an object independent from versions and the reference is immutable. Records that were once published in KCDC are frozen by versions, which means that even if the data are extended or changed the reproducibility is maintained.
In a next step we will adapt the data of a totally independent experiment
(TAIGA [
Data Publication