KOBE is a benchmarking system that leverages modern containerization and Cloud technologies. Data sources are formally described in more detail than what is conventionally provided, covering not only the data served but also the speci c software that serves it and its con gurations. KOBE provides a speci cation formalism and a command-line interface that completely hides from the user the details of provisioning and orchestrating the benchmarking process. Finally, KOBE automates collecting and comprehending logs, and extracting and visualizing evaluation metrics from these logs.
In the federated query processing community releasing a benchmark amounts to
releasing datasets, query workloads, and, at most, a benchmark-speci c
evaluation engine for executing the query load [
Research articles need to specify what software has been used to implement the SPARQL endpoints, how it has been con gured and distributed among hardware nodes, and the characteristics of these nodes and of the network that connects them to the federation system. Reproducing an experiment from such a description is a tedious task; Based on our own experience with federated query processing research we strive to minimize the e ort required and uncertainty involved in replicating experimental setups from the federated querying literature. ? Copyright (c) 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4/0).
Our rst step in that direction was to complement our own benchmark with
Docker images of the populated triple stores and of the federation systems [
Following up, we further developed the KOBE Open Benchmarking Engine into a framework that leverages modern containerization and Cloud computing technologies in order to reproduce benchmarking experiments. Data sources are formally described in more detail, covering not only the data served but also the speci c software that serves it and its con guration. KOBE provides a speci cation formalism and a command-line interface that hides from the user the mechanics of provisioning and orchestrating the benchmarking process on Kubernetes-based infrastructures; and of simulating network latency. Finally, KOBE automates the process of collecting and comprehending logs, and extracting and visualizing evaluation metrics from them. 2
KOBE comprises three subsystems (Figure 1): 1. The deployment subsystem uses Kubernetes to deploy containerized images of the experiment components, and deploys the operator that orchestrates benchmarking and monitors the progress of each experiment in the cluster. 2. The networking subsystem uses Istio, a Cloud-native controller that tightly integrates with Kubernetes. The KOBE operator utilizes Istio to setup the network connections between the data sources and the federation engine, explicitly controlling their quality to simulate a speci ed behaviour. 3. The logging subsystem is implemented as an Elasticsearch/Fluentd/Kibana (EFK) stack. It collects and manages the logs produced by the various experiment components (the data sources, federators and evaluators) and produces meaningful diagrams and graphs about the benchmarking process.
The users interact with the operator via a command-line interface to submit experiment descriptions. These descriptions are YAML les that specify the query load, the federator that is being benchmarks, and the data sources it federates. A collection of federators and benchmarks are included in the KOBE distribution, but the user is able to also provide their own as Docker images.
The operator uses the evaluator, provided as part of KOBE, to apply the query load to the federator. Istio manages the network connections between the data sources and the federation engine, allowing control over the virtual network characteristics. Finally, the EFK stack collects logs, extracts metrics, and provides visualizations of the these benchmark metrics. A set of Kibana panels relevant to benchmarking federated query processing are included in the KOBE distribution, but, naturally more can be de ned by the user. 3
A benchmarking experiment is expressed as YAML les that specify data sources, queries, and parameters of the experiment and the evaluator. The operator uses this information to deploy the necessary containers and execute the experiment. We will show the contents of these YAML les and demonstrate how their content impacts experiment outcomes. We will then show how to submit an experiment using the command-line application. We will inspect the deployed containers, showing how, for example, for metadata-based federation engines [e.g., Semagrow, 1] the relevant metadata has been extracted and made available whereas for engines where this is not required [e.g., FedX, 5] this step has been skipped.
We will also demonstrate the ability to separate the logs produced by multiple executions of the same query, e.g., to get an average or to try di erent parameters in di erent experiments. In order to di erentiate between executions and comprehend logs from a speci c execution of a speci c experiment, each log contains a unique id generated by the KOBE operator. The id is passed to the federator via a SPARQL comment. Federators that have been adapted to KOBE read this id and include it in the logs; those that have not been modi ed for KOBE are una ected as they ignore the comment in the SPARQL query.
Naturally, detailed metrics about each phase of query processing cannot be obtained from such engines, and results are only reported at the level of complete query runs. As an example, consider Figure 2 reporting an experiment execution for the life-science (ls) query set of the FedBench benchmark for the Semagrow federation engine (adapted to KOBE), and compare it to the coarser graph in Figure 3 that compare Semagrow and FedX (not adapted). 4
KOBE is a benchmarking system that allows federated querying researchers to publish more complete and reproducible descriptions of their experiments, and that automates the tedious tasks of deploying the data services needed to execute an experiment, collecting logs, and extracting evaluation metrics.
KOBE is developed as an open source project and the repository includes instructions and example experiments: https://github.com/semagrow/kobe
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825258. Please see http://earthanalytics.eu for more details.