This paper reports a use-case developed for an international IT company, whose one of multiple branches is located in Poland. In order to deploy a graph database in their IT architecture, the company needed an assessment of some of the most popular graph database management systems to select one that fits their needs. Despite the fact that multiple graph database benchmarks have been proposed so far, they do not cover all use-cases required by industry. This problem was faced by the company. A specific structure of graphs used by the company and specific queries, initiated developing a new graph benchmark, tailored to their needs. With this respect, the benchmark that we developed complements the existing benchmarks with 5 real use-cases. Based on the benchmark, 5 open-source graph database management systems were evaluated experimentally. In this paper we present the benchmark and the experimental results.
Among multiple database technologies [
When it comes to selecting a GDB to suit an eficient storage of given graphs and eficient processing, a company professional has to either implement multiple ’proofs of a concept’ or rely on existing evaluations of various databases. Typically, important assessment metrics include: (1) performance and (2) scalability w.r.t. a graph size and (3) scalability w.r.t. a number of nodes in a cluster.
In practice, assessing performance of IT architectures and particular software products is done by a benchmark. There exist multiple dedicated benchmarks for given domains of application. In the area of information systems and databases, the industry accepted and used benchmarks are developed by the Transaction Processing Council. There also exist dedicated benchmarks for non-relational databases and clouds, cf. Section 2.
Although there exist multiple benchmarks designed for graph databases, the motivation for our work came as a real need from industry, i.e., a large international IT company (whose name cannot to be revealed), having one of its multiple divisions located in Poland. The company stores large data volumes on various configurations of their software and network infrastructures. These data by virtue are interconnected and naturally form large graphs. Currently, these graphs are stored in flat files but in the future, they will be imported into a proprietary GDB and analyzed there. For this reason, a fundamental issue was to choose a GDBMS that would be the most suitable for particular ’graph shapes’ and queries needed by the company. The assessment criteria included: (1) performance characteristics w.r.t. variable number of nodes in a cluster as well as (2) functionality and user experience.
The specific structures of graphs produced by the company
and specific queries have not matched what was ofered by the
existing GDB benchmarks. These facts motivated the development
of the GoodBye benchmark, presented in this paper. Designing
GooDBye was inspired by [
The paper is structured as follows. Section 2 overviews benchmarks developed by research and industrial communities. Section 3 presents the benchmark that we developed. Section 4 outlines our test environment. Section 5 discusses the experimental evaluation of GDBs and their results. Finally, Section 6 summarizes the paper. 2
The performance of a database management system is typically
assessed by means of benchmarks. Each domain of database
application incurs its own benchmark. A benchmark is characterized
by a given schema (structure of data) and diferent workload
characteristics (query and data manipulation), and often by
performance measures. Database benchmarking over years have
received a substantial attention from the industry and research
communities. Nowadays, the standard industry approved set of
benchmarks for testing relational databases is being ofered by the
Transaction Processing Council (TPC) [
[
Benchmarking of other types of databases, like XML (e.g.,
[
In the area of GDBs, several benchmarks have been proposed
so far. [
[
[
[
[
The GooDBye benchmark includes: (1) a parameterized graph data generator, (2) a graph database, and (3) queries that are to be run on it. In order to use the benchmark, a user needs to: (1) run a data generator, (2) decide which GDBMS is to be tested, and install it on a cluster, (3) transform data generated by the benchmark into a form readable by the selected GDBMS, (4) load the data into a GDB, using its proprietary tool, (5) turn of database’s caching mechanisms, as the same subset of queries will need to be repeated multiple times, (6) run queries on the GDB. 3.1
A graph used in the benchmark is directed and cyclic, with a maximum cycle length of 2. The graph reflects typical software and hardware configurations in a large company. A node represents one of the three following data entities: • a package - it is composed of objects; a package can be transformed into another package; all packages have the same structure (fields); • an object - it is composed of fields; an object can be transformed into another object, similarly as a package; • a field - a field can be transformed into another field, similarly as an object, all fields have the same simple elementary datatype.
An arc represents: • a data transformation - packages can be transformed into other packages, objects into other objects, and fields into other fields; each transformation (identified by its ID) is represented at all three levels of data entities; • a data composition - each package contains one or more objects, and each object contains one or more fields.
The data generator is parameterized and can produce graphs described by diferent statistics. For the benchmark application presented in this paper, the graph had the following statistics: • the number of vertices: 911034, which represented 500 packages; • the number of arcs: 3229158, • the average number of objects in a package: 100 (binomial distribution n=8000, p=0.0125), • the number of object categories (types): 2; 30% of objects belong to category A, 70% belong to category B, • the average number of fields of objects in category A: 30 (binomial distribution n=1500, p=0.02), • the average number of fields of objects in category B: 8 (binominal distribution n=400, p=0.02), • the average number of incoming fields transformation arcs: 2.5 (binominal n=80, p=0.03125), • 4% of arcs form single-arc cycles, • 2% of arcs form two-arc cycles. 3.2
Eight queries were defined and implemented in the benchmark, as required by the company. Queries Q1-Q5 (described below) were demanded by the company. Q1-Q3 aim at checking how long it takes for a GDB to find neighbor vertices, as well as navigating via incoming and outcoming arcs. Q4 and Q5 check how fast a GDB finds nodes, having given in- and out-going arcs, to calculate an impact of changes on transformations. Q6-Q8 are typical queries that are defined in other benchmarks. • Q1 - it finds and returns all vertices that transform to node of type A, i.e., nodes that have an outgoing arc of type Transformation that is an incoming arc to A. • Q2 - it finds and returns all nodes that have an incoming arc of type Transformation, whose source is A. • Q3 - it counts all the vertices that are connected to a node by the Transformation arc lading to the node of type A and computes the percentage of these nodes over the number of all vertices in the graph. • Q4 - it counts all direct neighbors of nodes connected by a given transformation type and returns the percentage of the entire graph they comprise. • Q5 - it counts all direct nodes connected by a given transformation type, including nodes adjacent to nodes of type A. • Q6 - it counts the number of incoming and outgoing arcs of every single node in the graph, and returns a total count for each of them. This models a degree calculation for the entire graphs. • Q7 - it returns all nodes in the database that have their attribute equal to given number.
• Q8 - it computes the shortest path between two nodes. 4 4.1
The company imposed the following requirements on a GDBMS:
• to be run on either an open-source or academic licence,
• to be used in practice by industry (listed on the DB-Engines
website [
Based on the aforementioned criteria, out of 29 available GDBMSs, the following were selected for evaluation: ArangoDB, TitanDB, JanusGraph, OrientDB, and Spark GraphX. One system we heavily considered using was Neo4j. According to DB-Engines ranking, at the time of writing this paper, it was the most popular GDBMS. Unluckily, we were unable to obtain an academic license of its full, ’enterprise’ edition, providing among others distributed storage and processing. As such, we have decided not to test it, rather than unfairly assess its toy version. 4.2
The GDBMSs were installed in a micro-cluster composed of 9 physical machines. Each node was run by Ubuntu and had the following parameters: (1) 8GB RAM, (2) 457GB HDD, (3) Intel Core2 Quad CPU Q9650 3.00GHz, (4) graphic card: GT215. The machines were physically fully interconnected with each other, enabling direct communication whenever required. The logical connections depended on the database system used.
Depending on an experiment, there were 1, 3, 5, or 9 nodes used at any given time. Such cluster sizes allowed to use with 1, 2, 4, and 8 worker nodes, and 1 access/coordinator node. Data were partitioned as equally as possible between nodes, using data distribution mechanisms provided by each GDMS. 5
The goal of the experiments was to evaluate response time of the 8 queries outlined in Section 3.2, for the 5 GDBMSs under test. Each query was run twelve times on the same dataset, on 1, 3, 5, and 9 nodes. The highest and lowest measurements were discarded. An average value and standard error were calculated for the remaining measurements. Due to huge diferences in performance between the tested GBDMSs, a logarithmic scale was used in charts. Below we discuss the obtained performance characteristics. 5.1
The response time (elapsed) for Q1-Q8 was measured in milliseconds. Below we present the results and discuss the obtained performance characteristics.
sponse times for Q1 are shown in Figure 1. As we can observe, ArangoDB clearly outperforms all the other GDBMSs. GraphX is the only system for which query response time decreases with the increasing number of nodes. The performance of TitanDB and JanusGraph degrades with the increasing number of nodes - queries run on the 9-machine cluster take about twenty times more than when running on a single node.
1x107
The response times of this query are shown in Figure 2. In this test, ArangoDB, once again outperforms all the other GDBMSs, although the degradation of its performance when increasing the size of the cluster is more noticeable. GraphX execution times decrease by a factor of three, when a cluster size increases to 9, resulting in better execution times than TitanDB or JanusGraph, but still worse than OrientDB.
1x107
Q3 - measuring the impact of changes in a node. The response times of this query are shown in Figure 3. From the chart we can notice that ArangoDB has a clear lead over its competitors yet again, with its executions taking thousands times less than of the other GDBMSs. GraphX slowly approaches ArangoDB execution times as the cluster size increases. OrientDB achieves better results than TitanDB and JanusGraph on clusters of greater size. 100 10 1 1x107
ArangoDB 1x106 OTriiteanntDDBB ]s 100000 JanusGraph m GraphX [ item 10000 seon 1000 p s e r 100 10 1 1x107
ArangoDB 1x106 OTriiteanntDDBB ]s 100000 JanusGraph m GraphX [ item 10000 seon 1000 p s e r 100 10 1 1n 3n 5n
9n
tion. As Q4 query is very similar to Q3, the execution times shown in Figure 4 have characteristics similar to those shown in Figure 3, i.e., ArangoDB achieves the best results, with GraphX slowly decreasing ArangoDB lead as the cluster grows. 1n 3n 5n
9n
Q5 - measuring the impact of changes in topology. The execution times from this experiment are shown in Figure 5. Once again, ArangoDB is in the lead. GraphX execution times decrease as the cluster size increases. OrientDB performs worse that TitanDB and JanusGraph for a cluster size up to 3 and performs better when the cluster grows.
1n 3n 5n
9n
times from this experiment are shown in Figure 6. For this query, GraphiX ofers the best performance, regardless of the cluster size. In a 3-, 5-, and 9-node cluster ArangoDB performs the worst. The performance of OrientDB remains unchanged regardless of the cluster size.
1x107 1x106
Q7 - filtering query . The results of this evaluation are shown in Figure 7. On a single node, average execution times of the same query on ArangoDB and GraphX difer only by forty-five milliseconds, but as the cluster size increases GraphX gains noticeable lead over all the other GDBMSs.
1n 3n 5n
9n
ment was run on ArangoDB, OrientDB, TitanDB, and JanusGraph. The reason for eliminating GraphX was caused by the implementation of the shortest path algorithm in GraphX. Rather than simply finding the shortest path between two nodes, it finds all the shortest paths from all the nodes to the target one, only then allowing users to select specific paths from a generated RDD. This heavily influences execution times of such queries. The first stage (computation of the shortest paths) takes minutes rather than milliseconds, and the second (retrieval of specific paths) takes a few milliseconds, making the results fairly incomparable to other GDBMSs. Figure 8 reveals that ArangoDB handles this query in the least amount of time. On a single node, OrientDB performs worse than TitanDB or JanusGraph, and performs better on 3, 5, and 9 nodes.
In Figure 9 we present total execution times of a workload composed of queries Q1, Q2, ..., Q7, for ArangoDB, OrientDB, TitanDB, JanusGraph, and GraphX in a cluster composed of 1, 3, 5, and 9 machines. As we can observe, ArangoDB, TitanDB, 1n 3n 5n
9n and JanusGraph do not ofer scaling out, as the total execution time grows with the increase of the number of machines. On the contrary, OrientDB and GraphX ofer rather constant execution time w.r.t. the number of machines.
3n 5n
9n • ArangoDB achieves better results for Q1-Q5 and Q7-Q8 than GraphX, • ArangoDB achieves worse performance than GraphX for
Q6, • GraphX achieves better performance than OrientDB for Q6, since OrientDB is more eficient than ArangoDB in executing Q6.
To this end, we applied T-Student tests with p=0.01. The results of the significance tests are included in Table 1. The p-values for the significance of the results between GraphX and ArangoDB are represented by rows with queries Q1-Q5 and Q7, whereas p-values for the significance between GraphX and OrientDB are represented by row with Q6. Each row includes p-values for the experiments on 1, 3, 5, and 9 nodes.
1x107
As it can be seen, ArangoDB wins in this regard as well. Its installation is straightforward, setting up a cluster requires nothing but running a few, simple scripts. Its querying language is robust and intuitive, with a focus on sub-querying. It runs on most common operating systems. Its visual interface is decent.
TitanDB and JanusGraph are next in our ranking. Their installation and setting up a cluster require a bit of fiddling, although it does not require all that much skill. The query languages are easy to learn and use. Both of these GDBMSs do not expose any problems of running on any of the popular operating systems. They do require quite a lot of work to set up any kind of visual interface.
OrientDB scores third. Its installation is not dificult, although having a few instances in a cluster is problematic. Its language lacks a few built-ins. It supports numerous OSs. Visual representations of graphs it generates are decent and legible.
GraphX scores last. Ease of use was never the focus for Sparkbased tools. Installation and cluster set up is rather easy, but connecting it to a resilient data storage is more dificult. Tutorials for GraphX are almost non-existent, and documentation occasionally leaves a bit to be desired. Since it is Java-based, it has no problems running virtually anywhere. Graphical interface (other than Spark management tool) is nonexistent.
In this paper we presented a graph database benchmark developed to meet specific requirements of an international IT company. Even though over 10 graph benchmarks have been proposed in the research literature, none of them reflects the particular structure of the graph or particular queries needed by the IT company. Therefore, the benchmark that we developed can be considered as a complementary to those mentioned in Section 2. It contributes another graph structure used by industry and five queries used by industry.
The benchmark was implemented and used in practice to asses the performance of 5 open-source GBDMSs in a micro-cluster composed variable number of physical nodes (up to 9 nodes were used). The experiments that we run showed that: • distributing graph data into multiple nodes does not provide scaling out; we observed that: (1) query execution times increased when the size of the cluster increased (the case of ArangoDB, TitanDB, and JanusGraph) or remained approximately constant (the case of OrientDB and GraphX); • even simple queries can take much longer to execute in a cluster when a GDB needs to cross-check every node for arcs leading to another shard; • ArangoDB ofers the best performance in the majority of tests; it also ofers the best functionality from a user perspective; • GraphX ofers the best performance when it comes to massive localized data processing (cf. Figure 6), i.e., it is a good match for certain algorithms such as PageRank, that are highly interested in degrees of nodes.
The performance evaluation can further be extended to test the scalability of GDBMSs w.r.t. graph size and clusters of sizes greater than 9 nodes. To this end, the proposed GoodBye benchmark needs to be further extended as well, to generate graphs of parameterized size and multiple statistical properties.