Multi-model database systems have gained increasing popularity due to their eficient management of diverse types of data and support for complex queries. They ofer a unified approach for managing data in various formats, including structured, semi-structured, and unstructured data. However, benchmarking the performance of such systems is a challenging task, given their complexity, mainly due to their support for multiple data models. While significant research exists for benchmarking single-model databases, a comprehensive approach for evaluating multi-model databases is still in an early stage. To address this challenge, we propose MMSBench-Net, a benchmark for evaluating multi-model database systems that support structured relational, semi-structured document, and graph data models. MMSBench-Net enables comparative analysis of database systems and demonstrates how diferent workloads can reveal the strengths and weaknesses of multi-model database systems. To demonstrate the efectiveness of the benchmark, we compare the performance of two database systems: Polypheny and SurrealDB. Our work is a first step towards a comprehensive evaluation methodology for multi-model database systems.
CEUR ceur-ws.org
The field of data management has experienced a
significant transformation in recent years. While relational
cialized systems have emerged. Two data models that
have gained substantial popularity are the graph and the
document model. These data models allow data to be
represented and queried unknown from the relational
model [
While there are well-established
benchmarks like TPC-C [
This paper makes two contributions: Firstly, we propose a benchmark called MMSBench-Net that is tailored SurrealDB2 and discuss the results. based on a real-world scenario that deals with relational, document and graph data. Secondly, we demonstrate the utility of our benchmark by comparing the performance of two multi-model database systems, Polypheny1 and
The remainder of this paper is structured as follows: the underlying scenario and present the data, and workload that is being generated. In Section 3, we then briefly introduce the two multi-model database systems subject to the benchmark evaluation presented in this paper. Section 4 then presents and discusses the obtained results. The paper concludes with an overview of related work in Section 5, an outlook towards future work in Section 6 and a conclusion in Section 7.
To evaluate the performance of multi-model database systems, we propose MMSBench-Net, a benchmark that assesses their ability to manage structured relational, Bench-Net is designed to evaluate the eficiency and versatility of multi-model database systems under diferent workloads. The “-Net” sufix refers to the first scenario use cases that could be modeled optimally with the re- In Section 2, we introduce the MMSBench-Net, discuss (H. Schuldt)
Logs deviceId <number> error: optional ↳ message <string> ↳ type <string> ↳ stacktrace <array> users <array> optional message <string> optional timestamp <string>
[Device] [Connection] [Device] [Connection] [Connection] [Device] id: <int>, add. props
accesstime <timestamp> PK
id <int> PK deviceId <int> userId <int> FK duration <float> successful <boolean> firstname <string> lastname <string> birthday <int> salary <int> introduced in this paper. We plan to add more scenarios (and thus sufixes) in the future, leading to a complete suite.
The MMSBench-Net benchmark consists of a set of queries that reflect real-world use cases across the three data models. These queries are designed to evaluate various aspects of multi-model database systems, including their ability to handle complex data structures, support complex queries, and eficiently execute transactions. 2.1. Scenario deviceId: 3, timstamp: "2017-07-23:14-03", error: { message: "Out of Memory", type: "Application Error", stacktrace: ["Error on start of..."] }, user:{ id: 34, status: "logged in" }, users: [34, 45] MMSBench-Net is inspired by a real-world scenario of a company’s network monitoring application. Network monitoring plays a vital role in identifying and address- Figure 2: Example Status Log Showing an Error ing potential issues, threats and vulnerabilities in the network infrastructure, ensuring smooth operations and preventing data breaches or downtime. The monitoring its “purchase year” and other relevant information. application continuously collects all kinds of information In irregular intervals, each device produces a semiabout the network, including logged-in devices, usage structured log-entry containing information about its statistics and log messages, resulting in huge amounts of current state. An example of such a log can be seen in heterogeneous data. Figure 2. These logs might contain error information, in
The network monitoring application modeled by MMS- dicating problems with the device. All log entries include Bench-Net maintains data in three data models, (i) a the properties deviceId, timestamp and users. However, graph part, modeling the topology of the network, (ii) a additional properties with varying levels of nesting are document part, which consists of semi-structured logs randomly generated for each log entry. produces by the devices and (iii) a relational part, which An important information for monitoring a network holds basic information about the users and recorded is which person is currently associated with which dedata about their access patterns. The complete schema is vices. For this scenario, we assume a rather simple user depicted in Figure 1. database represented as a relational table containing in
The topology of the network is saved as a graph, where formation on the employee. Furthermore, there is also a each device in the network is represented as a node, and table for recording successful and failed login attempts a network connection between two devices is modeled and for accounting the usage of devices. Hence, this as an edge. For both, the devices modeled as nodes and scenario necessitates the database system to deal with the connection modeled as edges, additional information heterogeneous read and write workloads. is being stored, such as, the “manufacturer” of a device, 2.2. Schema and Data Generation To generate the schema and to populate it with realistic, but artificially created data, MMSBench-Net starts with building a simulation. This simulation includes the 2.3. Workload Generation graph representing the network that is being monitored, A workload consists of a collection of randomly chosen as well as the users interacting with it. The nodes in the queries according to a configurable distribution. Since graph represent devices (e.g., computers, mobile phones, the order in which queries are executed can impact the switches, and routers). The edges between these nodes performance of a database system (e.g., due to concurrepresent network connections between these devices. rency efects and locking), the implementation needs to The simulation utilizes the defined topology to gener- make sure that the workload is identical for all systems ate meaningful workloads. By making changes to this under evaluation (e.g., by using the same seed). MMStopology, it becomes possible to adjust the distribution Bench-Netuses a variety of queries to build its workloads: of available targets for the queries. This enables to easily align with specific requirements and desired focus of a Read Device or Connection Selects a device or conworkload. nection and retrieves it partially or fully. One of
The process of generating this simulated network con- the static parameters is chosen for this. sists of multiple steps: relational queries. The collected queries are then sequentially executed on the database systems.
User Generation First, a configurable number of users is being generated.
Read Log Selects a device and reads all or parts of its logs. Filters as well as projections of underlying keys are chosen from the target device.
Generation of Devices For each type of device (e.g., Remove Device Selects a device and deletes it, also switches, computers), a random number (within a all connections to this device are deleted as well. configurable range) of devices is being generated. Logs are deleted as well, information on log-in attempts are kept.
vice, a random set of properties is being generated. Remove Connection Randomly selects a connection Furthermore, a set of login, as well as status logs between two network devices and deletes it. is added as well.
of the network, multiple pairs of devices are selected and connections between them are created.
devices, connections do not have status logs, but they also create multiple dynamic properties.
Add Device Adds a device to the network. Generate
new connections to existing devices.
Remove Logs Randomly selects a device and deletes
some of its logs.
Add Logs Creates a random log message and adds it to existing devices or connections.
After the generation of the network is done, it is used as a template to create the workload. Each workload consists of queries in a query language supported by the Remove User Randomly selects a user who is being system under evaluation. deleted.
A distinction is made between the three data models.
First, the graph data is handled as already seen in Figure 1. Change User Randomly selects a user and adjusts an For this, each device is represented as a node and each attribute. connection is translated to an edge which connects them.
The small set of dynamic properties is inserted directly Besides simple queries, there are also more complex as part of these nodes and edges (if properties are not retrieval operations which can be chosen, their frequency supported by the data model, these are handled as if they is also configurable. are unstructured data). Then all generated device logs (an example can be seen in Figure 2), are translated to Connectivity Checks “Find all similar connected dedocument queries. Each entity of type device translates vices” or “Find connected device of specific type” its nested status logs into multiple document queries, Error Analysis “Identify the top 10 most common errors” each containing a timestamp and the ID of the device. or “Calculate the percentage of errors caused by
Finally, all login records are collected from the devices each user” and together with the user data itself are translated into Add User Creates a new user. All attributes are randomly generated.
Login Activity “Successful logins by user and month” or provide fast performance while adhering to these guaran“Average duration of successful logins by user and tees. It also supports unstructured data and basic graph hour of the day” functionality, which makes it a suitable choice for this comparison. SurrealDB was designed with the goal of re
Firstly, the actions are selected and implemented on the ducing the number of joins required for retrieval queries. simulated network, while concurrently being captured It accomplishes this objective by utilizing a graph strucand converted into queries for the evaluated systems. ture that allows a tuple to any other tuple. SurrealQL, a Once the simulation concludes, the gathered queries are SQL-like query language, is the primary means of interdistributed across a configurable number of available acting with the system, which can be accessed through threads and executed on the evaluated system. The exe- either a REST or a web socket interface. cution time for each query is measured individually and recorded for subsequent analysis. This facilitates a comprehensive analysis of various aspects of the database 4. Evaluation systems. Each iteration of this workload generation and execution process is referred to as a cycle; in an evaluation, multiple cycles can be chained together to construct more extensive workloads.
Our evaluation uses Chronos [
SurrealDB is a multi-model database management system that provides traditional database guarantees, such as ACID transactions, persistent data storage, and finegrained data access control. Its primary objective is to 3https://hsqldb.org/ SurrealDB Polypheny SurrealDB )sn 1013 , e l a c s g l(o 1012 e m i t n u lR 1011 a t o T ) s n , e l a c s g o l( e 1011 m i t n u R l a t o T
Polypheny SurrealDB Polypheny SurrealDB 10 20 30 40 50 60 70 80 90 100
Cycles 1x 2x
3x Device Scaling 4x 5x
However, in most real-world scenarios, the network
starts with a significantly higher number than the 10 One of the first prominent benchmarks for evaluating
users used by default. Thus, the number of users is being database management systems was the Wisconsin [
The results obtained from the evaluation of the two single-model databases, using a proposed synthetic
generquite diferent systems confirms the concepts of the MMS- ated benchmark. They were able to show, that depending
Bench-Net benchmark, in particular that it is agnostic to on the scenario, multi-model databases can be faster than
the concrete database under evaluation and has a wide configurations combining multiple single-model database
applicability for the evaluation of single- and multi-model systems. UniBench [
Our goal for MMSBench-Net is to extend it into a benchmarking suite that ofers various real-world usage scenarios for multi-model data management. However, there are some limitations with MMSBench-Net that we need to address in the future.
First, the minimal set of queries that we have chosen for evaluation may not be representative of all possible ways to query multi-model systems. In future evaluations, we should include a more diverse set of queries to reflect the range of possibilities when querying these systems. This would provide more comprehensive results and strengthen the obtained conclusions.
Second, the current composition of workloads is too broad and general to allow for nuanced comparisons of multi-model systems. We need to create more finegrained workloads that focus on specific aspects of data models to capture the subtle diferences between these systems.
In addition to the limitations of the benchmark, our evaluation only compared two systems, leaving a lot of unexplored territory. Future evaluations should include additional systems such as ArangoDB and OrientDB to gain more insights into their performance. Although not all multi-model databases support the same data models, it is possible to use parts of unsupported data models or substitute them with other models to expand the range of systems that can be evaluated.
Lastly, we should consider evaluating configurations that use a combination of multiple single-model databases to facilitate interesting comparisons. By addressing these limitations, we can develop a more comprehensive and nuanced benchmarking suite that ofers a more accurate evaluation of multi-model systems.
In this paper, we introduced the MMSBench-Net, a new benchmarked superficially tailored to benchmark multimodel database systems that is based on the scenario of a network monitoring application. Our evaluation of Polypheny and SurrealDB demonstrates the efectiveness and applicability of the proposed benchmark.
Our research represents an important first step towards establishing a comprehensive evaluation methodology for multi-model database systems. The proposed benchmark allows for a fair comparison of diferent systems, and our results provide insights into the performance of Polypheny and SurrealDB under diferent workloads. Ultimately, this benchmark will guide the development and evaluation of novel multi-model database systems.
This work was partly supported by the SNSF (“PolyphenyDDI: A Flexible Polystore-based Distributed Data Infrastructure”, grant no. 200020_213121). The authors would like to thank R. Arnold, R. Gasser, S. Heller, L. Sauter, F. Spiess and A. Mbilinyi for their valuable feedback.