Reproducibility is often mentioned as a core requirement for benchmarking studies of software systems and services. “Cloud-native” is an emerging style for building large-scale software systems, which leads to an increasing amount of benchmarks for cloud-native tools and architectures. However, the complex nature of cloud-native deployments makes the execution and repetition of benchmarks tedious and error-prone. In this paper we report on our experience with developing a benchmarking tool based on established cloud-native patterns and tools. In particular, we present a benchmarking tool architecture based on the Kubernetes Operator Pattern. Accompanied with a role model and a data model for describing benchmarks and their executions, this architecture aims to simplify defining, distributing, and executing benchmarks for better reproducibility.
Along with its growing adoption in industry, “cloud-native” [
However, we experienced that for cloud-native systems, this traditional way of providing benchmarks poses significant dificulties. Cloud-native applications typically consist of several processes, running in isolated containers, which have to be scheduled to diferent computing nodes. They require application-level configurations passed to these containers, comprehensive descriptions of internal and external network interconnection, and integration of diferent storage systems. These and other aspects make the deployment and operation of cloud-native applications a rather complex task and, hence, also complicate benchmarking.
On the other hand, several tools emerged in the cloud-native community for enhancing the
deployment and operation. For example, Kubernetes is nowadays the de-facto standard for
declarative orchestration, Prometheus collects monitoring data of various sources, and Helm
simplifies packaging, distributing, and installing entire applications. Recently, the Kubernetes
Operator Pattern [
After a brief introduction of Kubernetes and the Operator Pattern, we present the necessary data model and system architecture for a benchmarking operator. We complement this paper by an overview of how we apply this approach in our Theodolite scalability benchmarking tool.
Kubernetes is a declarative orchestration tool for cloud-native applications. Users describe the desired state of an application in so-called resources and in a continuous reconciliation loop, Kubernetes adopts the current system to reach the desired state. A typical example of a resource is a Deployment, which describes an application component (e.g., a microservice) consisting of container images, number of replicas and others. A common way for users to interact with Kubernetes is by describing resources in YAML files and applying these files to Kubernetes by using the kubectl command line tool.
The Kubernetes Operator Pattern [
As presented in Section 2, the Operator Pattern combines knowledge of operating Kubernetes with domain knowledge. As the domain at hand is benchmarking, we propose that actors involved in benchmarking describe their operational knowledge in CRDs. In general, we can observe two actors involved in benchmarking: The Benchmark Designer has knowledge about operating a system under test (SUT) as well as how the quality to be benchmarked can be measured (metric and measurement method). Additionally, he/she defines how metrics and results can be interpreted to compare diferent SUTs. The Benchmark Designer bundles all of this in a benchmarking tool or sometimes in an artifact that can be interpreted by a benchmarking tool. The Benchmarker intents to compare and rank diferent existing SUTs, evaluates new methods or tools against a defined standard, or repeats previous experiments. For this purpose, publishes to Benchmarking
Operator publishes to
Benchmark
Repository Research Data
Repository Benchmark Designer creates
Benchmark refers to deploys to deploys to
he/she executes existing benchmarks. A detailed description of this actor can be found by
Kounev et al. [
Based on this distinctions of roles, we propose a data model in which benchmarks and their executions are separated. Figure 1 illustrates this. We envisage individual CRDs for both benchmarks and executions: Benchmarks are defined by Benchmark Designers. They describe the SUT as well as the load generation. Additionally, they may describe configurations of the benchmarking method, which are interpreted by the benchmarking tool. Benchmarks can be published as supplemental material of research papers, but ideally they are versioned and maintained in public repositories (e.g., at GitHub). Benchmarks are stateless as they can be executed arbitrarily often.
Executions on the other hand describe a single execution of a Benchmark. They refer to exactly one Benchmark and configure the experimental setup. Definitions of executions can be shared, for example, as part of a research study that benchmarks the performance of diferent SUTs. When deployed to Kubernetes, Executions have a state, which is typically something like Pending, Running, Finished, or Failed if an error occurred (similar to Kubernetes Jobs).
Standard software components involved in benchmarking are a system under test (SUT), a load
generator, an experiment controller, a measurement database, and an (optional) monitoring
component for passive observation [
Kubernetes API
Operator notifies updates states benchmarker will interact with such a benchmarking tool.
To execute benchmarks, a Benchmarker solely interacts with the Kubernetes API, for instance, via tools such as kubectl or the Kubernetes dashboard. The Benchmarker registers benchmarks and initiates their executions by applying the corresponding CRs. The operator continuously observes the Kubernetes API for changes in the list of registered executions. It takes on the role of the experiment controller and decides based on a tool-specific policy whether it starts an execution immediately or schedules it to a list of pending executions. A typical constraint is, for example, that only one execution should run at a time to avoid interfering benchmark results. Once a registered execution is started, the operator reads the CRs of that execution and the corresponding benchmark. It creates and starts the SUT and the load generator via the Kubernetes API as well as performs additional cluster configurations.
While executing the benchmark, the operator updates the state of the Execution (e.g., to a state Running) and registers events (e.g., once a certain phase of the execution is completed). State and events of an Execution are managed by the Kubernetes API and are used to provide feedback to the Benchmarker. During the entire benchmark execution, Prometheus collects monitoring data from the SUT and the load generator. This can often be done in the same way as the cloud-native applications is monitored in a production deployment. Once a benchmark execution is finished, the operator stops and removes the SUT and the load generator. In addition, it requests collected monitoring data from Prometheus and analyzes them. Also requesting and analyzing monitoring data during an execution to adjust the SUT and the load generator is possible, which is necessary for certain types of benchmarks.
The operator stores benchmarking results persistently (e.g., in a database or as CSV files in a Kubernetes volume), from where the Benchmarker can access them for ofline analysis, archiving, or sharing. For the passive observation of benchmark executions, Grafana can be integrated in the architecture to visualize the monitoring data in real time.
Installing a benchmarking tool based on our proposed architecture might quickly become a complex tasks due to the many components and dependencies involved. In particular for reproducibility, it is often also necessary to install a specific version of a benchmarking tool. To we expect the mentioned tools to be rather matured and that possible successors can be used similarly. simplify installation and upgrading, we propose to package an operator-based benchmarking tool with Helm. Helm then registers the benchmarking CRDs and installs the operator with all its dependencies.
Theodolite [
The Benchmark resource specifies the SUT and the load generator as sets of other resources,
which describe their deployment. Since scalability is measured according to a load type and a
resource type [
The Execution resource refers to a Benchmark resource by its name. It selects one load and
one resource type from those provided by the Benchmark and specifies a set of values to be
evaluated for each type. Theodolite performs isolated experiments for a certain load intensity
with a certain amount of resources and checks for each experiment whether SLOs are met
[
The Kubernetes Operator Pattern allows to manage application-level configuration as part of the orchestration process. Applied to benchmarking cloud-native applications, it allows managing benchmarks and executions directly via the Kubernetes API. Main benefits of a benchmarking operator are: • Benchmarking tool, benchmark, and execution are clearly separated. 2https://github.com/cau-se/theodolite/blob/master/theodolite/examples/operator/ • Benchmarks are defined according to a strict format and published as descriptive files without including any tooling. • All experimental setup is bundled in one file describing the execution of a benchmark. • Benchmarks and Executions are created in a purely declarative way, saving Benchmark
Designer and Benchmarker from implementing any Kubernetes-specific error handling. • Benchmarks and Executions can be managed via default Kubernetes tools (e.g., kubectl), providing a rich user experience without additional implementation efort. • Since their CRDs can have an OpenAPI schema, Benchmarks and Executions can directly be validated by the Kubernetes API.
All in all, we expect building a benchmark tool with the operator pattern facilitates repeatability and, hence, ultimately reproducibility of cloud-native benchmarking studies.