High-energy physics experiments, such as those being conducted at the Large Hadron Collider (LHC) at CERN and will be conducted at the Nuclotron-based Ion Collider fAсility (NICA) at JINR, produce actual experimental data at the scale of terabytes per second [1]- [3]. This data is usually processed and analyzed using specialized libraries on dedicated computing platforms [4]. In addition, modern large experiments and institutions generate many streams of ancillary data that plays a critical role in supporting their operations. This information has an immediate technical purpose, but it can also be collected for a more thorough cross-referential analysis.
Projects in the Big Data ecosystem provide robust and scalable software to build a platform for collecting and processing such datasets. A prototype of such a platform was proposed and implemented in [5]. This work aims to build on the given progress by implementing support for GPU computing resources. The speedup that GPU processing ensures for different processing and analysis operations can be then measured for more effective scale-out and task scheduling in the future. We expect GPUs to be especially effective for accelerating the training of machine learning models built with deep artificial neural networks.
The core processing and analysis framework of the platform is Apache Spark, which facilitates batch and stream processing, contains machine learning libraries, and can interface with many data management and storage tools in the Big Data ecosystem.
Distributed storage is provided within the platform by the MooseFS file system. This does not give the performance benefits of data locality afforded by HDFS, which stores data directly on compute nodes. However, data locality is reported to be less essential for modern Big Data platforms than it was at the inception of the Hadoop ecosystem [6].
The Spark cluster can be run standalone, or use a resource manager: YARN, Mesos or Kubernetes. Mesos was used as the resource management tool of the prototype framework in [5], however, Spark does not yet support GPU resource management with Mesos. Kubernetes was selected as the resource manager for the future, as it allowed consolidating the management of the computing resources and the containerization of platform services.
Running Spark in the standalone cluster mode is straightforward, but it is less flexible and less desirable in a production environment than the other modes. YARN is a resource manager specialized for the Big Data ecosystem; it would be preferable if we had an established Hadoop-based platform to add Spark onto.
To use NVIDIA GPUs as resources in Spark jobs running on the Kubernetes cluster, the underlying containers need to support NVIDIA hardware. The libraries and tools provided by NVIDIA for this support are multi-layered [7]:
• libnvidia-container provides an API and CLI to set up containers with NVIDIA GPU support • nvidia-container-toolkit provides a runC prestart hook to apply these compatibility tweaks on container startup • nvidia-container-runtime wraps runC, adding this prestart hook to any container config started through this wrapper • nvidia-docker2 installs the runtime into the local Docker configuration, allowing to start GPUenabled containers more easily The actual need for these tools and the compatibility between Kubernetes and the NVIDIA driver version is not very well-documented. Kubernetes suggests using k8s-device-plugin, which purportedly requires a specific NVIDIA driver version (384.81) [8], [9]. NVIDIA themselves provide more up-to-date and complete documentation on installing a Kubernetes cluster with GPU support [10]. We used the NVIDIA DeepOps approach suggested by that article. It provides an easy way to deploy and configure most of the Kubernetes components needed to run a production cluster by building on top Kubespray for Kubernetes deployment with Ansible [11]. The specific up-to-date procedure allowed us to quickly deploy and test the cluster for our platform, but it can create support and configuration issues in the future when we might want to deviate from the suggested cluster architecture.
An important aspect of managing GPU resources with Kubernetes for Spark is resource discovery. That is, finding and annotating the GPUs present on the Kubernetes node to direct specific Spark jobs to utilize such resources. DeepOps configured the containerized service for resource discovery by default.
To test GPU resource support on the platform, a sample distributed machine-learning job was run on the platform. Spark-tensorflow-distributor [12] provides a method for distributing Tensorflow workflows across the Spark cluster, basically utilizing Spark as a resource and workload manager. It also provides a sample script to demonstrate the training of a small convolutional neural network for the classic problem of handwritten digit classification on the standard MNIST dataset. As Tensorflow can run with or without a GPU, the same test script was used for testing throughout this project to ensure that the GPU virtualization of the NVIDIA T4 GPU worked with the standalone Spark node, that the distributor library worked correctly on the Kubernetes cluster, and that GPU resources were available to Spark through the Kubernetes node. To demonstrate a more practical application of the platform at a scale closer to Big Data, a prototype pipeline to collect and process network packets was implemented on the new framework, in an approach similar to the way the same problem was solved on the prototype framework [5].
Network packets from one local laboratory subnetwork were duplicated and sent to one physical machine that did not take part in the main Kubernetes cluster. Raw packet headers were extracted with tshark [13] running in a Docker container and dumped into temporary 10Mb files continuously. They were parsed by running 7 tshark instances with GNU parallel [14] in another container, with the communication of files to be parsed managed by incrontab and a named FIFO pipe. The parsed JSON files were immediately compressed into ~14Mb zstd archives and stored on the distributed storage. This collection step ran for continuous capture for 7 days, resulting in a parsed dataset of 700Gb ready for analysis.
The analysis was carried out in Spark, with the processing steps submitted from the Zeppelin notebook-style web interface running entirely within the same Kubernetes resource cluster that runs the resulting Spark jobs. Thanks to building our own Docker images hosted on a private Gitlab repository, mounting the distributed storage into the Zeppelin and worker nodes, version compatibility, and adding support for reading zstd-compressed files were minor problems.
The analysis consisted of using the Numeric Aggregate and Mode (NAGM) method to aggregate and extract network node features from the network packet TCP and IP headers. This method, specifically for Darknet packet analysis, is described in detail in [15]. We apply it to normal network packets to test the performance of the framework and engineer network node features with an existing established method for further analysis. The same approach was used to test the prototype Big Data framework in [5].
Most of the platform changes from the 2020 prototype to the current state were motivated by the inclusion of GPU resources, which necessitated the change of the resource manager, the update of Apache Spark, the use of Ansible for initial deployment. A list of changes and the motivation for them are presented in Table 3.
The study was carried out at the expense of the Russian Science Foundation grant (project No. 19-71-30008).