We will introduce a data analysis extension for the KASCADE Cosmic-ray Data Center (KCDC), based on the Jupyterhub/notebook ecosystem. A user-friendly interface, easy access to data from KCDC, and modern analysis software are of special interest. This contribution will discuss the service architecture, followed by a brief usage example. The KASCADE Cosmic-ray Data Center (KCDC) [2] is a public data shop developed at the Institute for Nuclear Physics at the Karlsruhe Institute for Technology. KCDC provides access to reconstructed cosmic-ray data taken by the KASCADE [1] experiment. We identi ed improvements and extension that will bene t our users: { The data-sets can be very large, depending on the cut and selection criteria, and personal disk-space and bandwidth to download the desired data-set quickly becomes a bottleneck. { Analyses usually rely on specialised frameworks and toolkits, some of which are very cumbersome to nd, install, use and/or maintain. One obvious solution to this is, instead of bringing data to the researcher, we bring the researcher to the data.
2.1 In order to quirements.
Requirements
nd the best possible technology, we formulated the following re{ Accessibility: we want people to actually use it, not yet another prestige project. { Usability: the environment should provide all the of the tools necessary to conduct an analysis of KASCADE data. { Administration: the entire software stack should be easy to install, maintain and update.
{ Don't reinvent the wheel, most of the work has already been done. 2.2
Accessibility We decided to use the Jupyterhub / Jupyter Notebook [
Useability Python is the current go-to language for data analysis and there's
a plethora of third-party analysis packages. We also have to provide ROOT [
Administration Notebooks allow arbitrary code execution by design, so it is
sensible to isolate the notebook server of each user from one another and from
the operating system. It shouldn't come as a surprise, that docker [
Though there is some custom code needed to attach Jupyter to KCDC, the majority of components are o -the-shelf industry standards. It is unlikely that we have to heavily invest into in-house development to keep these running or even extend and improve over time. 2.3
Jupyter notebooks are a special kind of le format for cell-based, interactive programming. The standard user interface is implemented as a web service (notebook server); the important features in context of this document are the le-system-like contents manager and the notebook text-editor/execution environment ("kernel" in ipython jargon).
The notebook servers are docker containers created on demand with exclusive access by a single user each. They are proxied by another container running the jupyterhub software; the jupyterhub handles authentication and creation/deletion of notebook servers.
Choosing docker as virtualisation technology has many bene ts, rst and foremost process isolation, which is a good idea when your service requires arbitray code execution. Realizing the service as a docker stack, we not only gain easy scalability for many users across multiple physical host machines, but also a text-based infrastructure description, which makes deployment and version control trivial.
For simplicity, users can use their KCDC credentials for authentication.
The contents manager has been extended for transparent access to the KCDC download area, and it is comparatively simple to add even more data shops, depending of course on the interfaces exposed by a data shop.
The current default notebook is con gured with kernels for both Python and C++ (via the ROOT interpreter).
A graphical representation of the proposed analysis framework with KCDC
integration can be found at Fig. 1.
This section is generated from the ipython notebook presentation at DLC 2020 [
We also showed the importing of data from KCDC to the analysis framework in the original presentation, and although it is a key design aspect of the described framework, the ease-of-use of this feature doesn't lend itself to an enlightning description on paper, short of an uninspired screenshot or the mention of "There's a button for that".
Successfull acquisition of data is therefore presupposed in the following analysis example.
We are going to apply the KCDC example analysis to one of the preselected
data-sets, for reasons of recreatability. After importing both the data le and
analysis script, create a new notebook in the same directory.
[
,!extractall()
os.listdir()
[
KCDC analyze example.C , slides.ipynb , KASCADE SmallDataSample wA runs 0877-7417 ROOT.zip , info.txt , events.root , EULA.pdf ]
Along with preselected data-sets, KCDC also provides a basic example
analysis in C++ + ROOT. In order to run the KCDC example analysis, switch the
notebook kernel to C++ and enter:
[
KCDC-Entries read from files: 1080295 KCDCM-Entries: 1080295 Array Entries: 986577 Calor Entries: 250981 Grande Entries: 88259 General Entries: 1080295 KCDCN-Entries to be evaluated: 1080295 processing event No: 0 of 1080295 processing event No: 100000 of 1080295 processing event No: 200000 of 1080295 processing event No: 300000 of 1080295 processing event No: 400000 of 1080295 processing event No: 500000 of 1080295 processing event No: 600000 of 1080295 processing event No: 700000 of 1080295 processing event No: 800000 of 1080295 processing event No: 900000 of 1080295 processing event No: 1000000 of 1080295 Entries survived:: 1080295 out of 1080295 general id >0 :: 1080295 array id >0 :: 986577 calorimter id >0:: 250981 grande id >0 :: 88259 (int) 0
Switch back to Python, and use PyROOT to open and display the result.
[
ROOT.gPad.SetLogy()
f.Get(key).Draw()
c.Draw()
[
The result of this example analysis shows, some statistics of the data set (ex. events over run number), some spectra (ex. energy deposit -Detectors), among other interesting things.
We consider the integration of our data shop and analysis framework as a successfull technology demonstration. Going public and making it available for all KCDC users is the current goal, and we also plan to extend the functionality; it is comparatively simple to add more o -site data shops, there's the possibility to have multi-core and GPU support, and we need to evaluate more third-party analysis packages and notebook extensions. We also anticipate that user feedback is going to be another driving force behind long-term improvement of this service.