The KM3NeT neutrino detector, a water Cherenkov experiment to detect relativistic charged particles, is currently under construction at two deep-sea locations in the Mediterranean Sea. As crossdomain experiment between neutrino, astro-particle and astrophysics, data processing and data publication from KM3NeT draws on computing paradigms and standardization from all elds. In this contribution, key considerations for the provision of multi-purpose open data sets, interface options and interoperability requirements are presented.
The ARCA and ORCA detectors designed, constructed and operated by the
KM3NeT Collaboration for the detection of high-energy neutrinos, follow
identical technical design, data acquisition and processing principles. However, the
detectors aim for two main scienti c goals from di erent elds of physics. While
ARCA is intended for the detection of high-energy neutrinos from the cosmos,
spanning a wide range of astrophysical research topics, ORCA targets neutrino
oscillation research utilizing atmospheric neutrinos and is therefore rooted in
particle physics. The KM3NeT science cases interlink to a wide range of
fundamental physics, contributing e.g. to dark matter searches or multi-messenger
follow-ups of gravitational wave alerts, see [
Implementing open science standards in KM3NeT therefore is not only a necessity to ensure full scienti c exploitation of the generated data both inside KM3NeT and in the general physics community, it also requires a leveled and diversi ed approach to meet the respective needs of the various communities. This includes the generation of accessible and interoperable open science products, harmonizing data and software standards and an ongoing search for ? for the KM3NeT collaboration
Copyright ' 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). best-practice solutions to explore the technical, organizational and scienti c limitations of data sharing while aiming for maximum transparency of scienti c procedures and outcomes. 2
The method to detect high-energy neutrinos in KM3NeT is based on the measurement of Cherenkov photons from charged particles generated in neutrino interactions in or around the instrumented volume of the detector in the deep sea. Pressure-resistant glass spheres, so-called digital optical modules (DOMs), each containing 31 photomultiplier tubes, register individual photons with timing precision on the nanosecond scale. In its nal con guration, each building block will consist of 115 detection lines with 18 DOMs each, transferring all instrument readout, especially PMT signals above a pre-set threshold, to the shore station. As a single DOM sends digitized PMT readouts and control data at a rate of up to 100 Mb s 1, this continuous data stream needs to be heavily reduced before further processing and storage. In the nal KM3NeT con guration, two building blocks for ARCA and one building block for ORCA with denser spacing to target lower-energy GeV-scale neutrinos are planned.
Data processing therefore follows a tier-based approach [
For physics analyses, derivatives of these enriched data sets are generated and their information is reduced to low-volume high-level data which can be analysed and integrated locally into the analysis work ow of the scientist, see Figure 1. For interpretability of the data, a full Monte Carlo simulation of the data generation and processing chain, starting at the primary data level, is run to generate reference simulated data for cross-checks at all processing stages and for statistic interpretation of the particle measurements. 2.1
Data processing at the DAQ level follows paradigms of particle physics and
utilizes computing and software methodological approaches of this community.
At the shore stations, event triggering in the Data Acquisition (DAQ) system
leads to a signi cant reduction of the data stream. The data stream also
includes relevant instrumentation readouts for a comprehensive understanding of
data taking conditions. Photon-related information is written to ROOT-based [
HPC clusters registered
HPC clusters any local computer
KM3NeT
Summary format reconstructed
particle properties event selectors data classification
parameters analysis params.
EOSC Open Formats celestial coordinates reconstructed
particle properties context information probability functions ...
KM3NeT internal format ● sim: “event” ● DAQ: ROOT tree-based Data level 1 Data level 2 KM3NeT internal format ● ROOT tree-based ● conversion to hdf5 Data level 3 KM3NeT high-level format ● ROOT tree-based ● hdf5
KM3NeT
KM3NeT
Data level 4 Community-based format ● VOEvent, VOTable ● KM3NeT hdf5 format ● use-case specific services detector particle simulation
triggering <vec> DOM channels instrumentation
<vec> Triggered hits reconstructed
particle properties event selectors description, user access rights and open data publication layer. high-performance computing (HPC) clusters. Instrumentation and environmental data collected at the detector site are stored separately in a central database. Acoustic and other environmental data serve as basis for Earth and Sea-science initiatives. Access to this information following an Open Science approach is under development, however, it will not be covered in the scope of this report.
Both the complex process of neutrino detection in a natural environment and the low expected count rate of the cosmic neutrino signal in comparison to atmospheric background events necessitates the full modelling of particle generation, detector response and data processing. To this end, a dedicated simulation chain, starting form cosmic air-shower particle generation or astrophysical neutrino ux assumptions, replicates the complete data-processing pipeline. At the event generation level, photon distributions induced by these particles within the detection volume are generated, masked by a simulation of the detector response and treated to the same processing as measurements starting from the second data level of the o ine event format. 2.2
Processed event data sets at the second level represent input to physics analyses, e.g. regarding neutrino oscillation and particle properties, and studies of atmospheric and cosmic neutrino generation. Enriching the data to this end involves probabilistic interpretation of temporal and spatial photon distributions for the reconstruction of event properties in both measured and simulated data, and requires high-performance computing capabilities. Due to the distributed infrastructure of the KM3NeT building blocks and the contribution of computing resources from various partners, data processing will, in the nal detector con guration, necessitate a federated computing approach, the implementation of which is prepared through containerization of the required software and testing of distributed resource management approaches. In this context, the use of a middleware like e.g. DIRAC1 is explored, again linking closely to the particle physics community.
Access to data at this level is restricted to collaboration members due to the intense use of computing resources, the large volume and complexity of the data and the members' primary exploitation right of KM3NeT data. However, data at this stage is already converted to HDF52 format as a less customized hierarchical format. This format choice increases interoperability and facilitates the application of data analysis software packages used e.g. in machine learning and helps to pave the way to wider collaborations within the scienti c community utilizing KM3NeT data. 1 Distributed Infrastructure with Remote
http://diracgrid.org/ 2 The HDF5 le format, https://www.hdfgroup.org/ Agent
Control
Interware, 2.3
Summary formats and high-level data As mostly information on particle type, properties and direction is relevant for the majority of physics analyses, a high-level summary format has been designed to reduce the complex event information to simpli ed arrays which allow for easy representation of an event data set as a table-like data structure. Although this already leads to a reduced data volume, these neutrino data sets are still dominated by atmospheric muon events at a ratio of about 106 : 1. Since for many analyses, atmospheric muons are considered background events to both astrophysics and oscillation studies, publication of low-volume general-purpose neutrino data sets requires further event ltering. Here, the choice of optimal lter criteria is usually dependent on the properties of the expected ux of the signal neutrinos and performed using the simulated event sets.
Event simulation derivatives as service To correctly judge the
statistical signi cance of a measured neutrino event rate, the full high-level simulation
data sets are used in KM3NeT internal studies to ensure a high accuracy of the
count rate estimate. As handling these large data sets is impractical for
interexperimental studies, but the information is crucial for the interpretability of
the data, parameterized distributions of relevant observables need to be derived
from the simulation data sets and o ered as services. Even in absence of
signi cant neutrino measurements in the construction phase of KM3NeT, o ering
sensitivity estimates as in [
The widely-accepted paradigm for open science data publication requires the
implementation of the FAIR principles [
The Virtual Observatory (VO) standards[
Neutrino sets in the VO Tabulated high-level neutrino event data can be provided through the VO registry, utilizing access protocols like the Table Access Protocol (TAP) and query languages like the Astronomical Data Query Language (ADQL). To query these data sets related to astronomical sources, the Simple Cone Search (SCS) protocol allows to pick speci c events according to particle source direction, using Uni ed Content Descriptors (UCDs) to identify the relevant table columns. The underlying data format is the VOTable which allows for metadata annotation of data columns. As the DaCHS software provides input capabilities for various formats like FITS6 or text-based tables on the server side, a common KM3NeT open event table format can be chosen quite independently and the interface adapted such that high-level neutrino data sets can be both o ered through the VO and alternative access protocols, as long as the required metadata description is handled adequately.
VO standards are at the current stage not fully adapted to the inclusion of neutrino data and require development of metadata standards for easy interpretability of the data, a matter which is targeted within the ESCAPE project. Open questions in this regard are the linkage of observation probabilities to a given event selection, the inclusion of \non-observation" in a given eld of view and within a given time as relevant scienti c information to retrieve from the services, and the introduction of a dedicated vocabulary for the description of neutrino data. This vocabulary will need to be developed within KM3NeT as a matter of internal standardization, however, the process will draw guidance from the VO expertise and framework. 3 see https://www.km3net.org/km3net-infradev/ 4 European Science Cluster of Astronomy & Particle physics ESFRI research Infrastructures, https://projectescape.eu. 5 at http://vo.km3net.de/ 6 Flexible Image Transport System, https:// ts.gsfc.nasa.gov/.
Multimessenger alerts Single or stacked neutrino events of high
astrophysical signal probability will be selected in KM3NeT to trigger an alert to other
observatories indicating a possible target for multimessenger observations [
Data modelling and access Publishing interfaces to data sets requires the generation of an accessible data model according to the W3C8 standards to well de ne both interface and data format, and will be provided alongside the KM3NeT data. For a basic KM3NeT open data format, the requirements on the format and access methods are building on current software solutions already in use within the KM3NeT collaboration. Interfacing to full tabular data sets beyond the VO requires the choice of a standard data format and the provision of a dedicated but easily usable software interface to this data. Here, a pythonbased approach is favoured due to the wide use of the related tools in the science community, with access to data sets ensured e.g. through the astropy package9 for FITS tables or python data analysis packages with HDF5 reader capabilities like pandas10. Usage examples and exemplary work ows for analyses will be provided as Jupyter11-notebooks.
Software requirements As KM3NeT software development also follows an Open Software approach, dedicated high-level analysis software is made available to facilitate proper handling of the research data. The software is provided 7 J. Swinbank, Comet, https://comet.transientskp.org. 8 https://www.w3.org/ 9 The Astropy Project https://www.astropy.org/ 10 see https://pandas.pydata.org/ 11 see the Project Jupyter https://jupyter.org/ as Docker12 or Singularity13 virtual containers on the respective KM3NeT registry server, with usage examples and source code made available through the KM3NeT Gitlab14 instance. The easy deployment of the software should enable users to locally analyse smaller data sets on their personal computer either in a containerized environment or by installing, if preferred, custom-made pythonpackages for KM3NeT data handling. Access to services and data sets will probably be implemented through the creation of a dedicated REST-API15 to the KM3NeT open data server and facilitated by providing dedicated function wrappers. This software development will also be integrated into the development of the Open Software and Service Repository in the ESCAPE project. Data access However, this approach is only feasible for small-scale data sets. Beyond this, access to external computing resources is required and computing needs to be moved to an integrated cloud environment allowing user authentication, easy management of the software environment and running of custom scripts on data stored by federated data providers. This transition from a locally restricted user level to integrated open science computing is currently under development for the European Open Science Cloud (EOSC), and the software choices for the KM3NeT Open Science environment as outlined above aim to integrate into this development. Here, containerized software will be made available, while larger data sets can be queried from the EOSC data lake, and the use of Jupyter notebooks for the use in an Open Science Portal is currently also envisioned. 4
KM3NeT data generation and publication draws from two di erent worlds of experimental physics by being rooted in particle physics especially in the generation of the data, while targeting both particle and astrophysics on the research level. Providing meaningful interfaces to research products like data sets and ux sensitivity estimates and instrument response functions can therefore not only rely on ready-made data sharing paradigms like the VO standards but holds the potential for new developments in order to make high-energy neutrino data FAIR in both worlds. Here, the ESCAPE and INFRADEV projects provide the ground for these common endeavors from both communities and the developing KM3NeT Open Science standards are integrated into this process. As the EOSC environment is also aimed at fostering interaction between scientists and co-development of projects and software, the scientist as user is here expected to become a potential partner in research and development, shifting the focus from 12 see https://www.docker.com/ 13 see https://sylabs.io/singularity/ 14 see Gitlab (https://about.gitlab.com/) at https://git.km3net.de/ 15 Fielding, R.T., Architectural Styles and the Design of Network-based Software Architectures, UCI, 2000 the sole providing of the data to the interaction amongst researchers. Therefore, both at the current development stage and in the future, KM3NeT Open Science rst and foremost builds on the knowledge-sharing and common ideas in the wider science community.