Although quite a number of RDF triple store benchmarks have already been conducted and published, it appears to be not that easy to find the right storage solution for your particular Semantic Web project. A basic reason is the lack of comprehensive performance tests with real-world data. Confronted with this problem, we setup and ran our own tests with a selection of four up-to-date triple store implementations - and came to interesting findings. In this paper, we briefly present the benchmark setup including the store configuration, the datasets, and the test queries. Based on a set of metrics, our results demonstrate the importance of real-world datasets in identifying anomalies or differences in reasoning. Finally, we must state that it is indeed difficult to give a general recommendation as no store wins in every field.
The last months inevitably reveal the advance of Semantic Web technologies
in organizing and finding information, e. g., through the advent of schema.org
or Google Knowledge Graph [
Although, a number of performance reports already exist, we soon discovered
that available results are of limited significance for our particular purposes. The
goal of our research project Topic/S [
A review of the existing work in the area of RDF store benchmarking [
In this paper we present the latest results of “yet another” benchmark of current RDF stores – but with the following unique features: loading and querying real-world datasets, testing RDFS reasoning and SPARQL 1.1 queries, conducting multiple queries in parallel, and recording the memory requirements.
Thus, the paper is organized as follows: In the next section, we briefly summarize the benchmark setting, including the selected RDF stores, the datasets and queries. In Section 3, we shortly introduce the metrics, present and discuss the results of our benchmark. A conclusion and outlook on future work is given in Section 4. 2
In this section, we briefly introduce which RDF stores we have selected and how we set them up in our benchmark. In the second part, we present the four real-world datasets and the queries we utilized for our evaluation. 2.1
Although, there are more RDF stores available, we focused on Apache Jena
[
The Apache Jena projects comes up with a bunch of sub-systems. For our purpose we needed a fast triple store as well as a SPARQL endpoint, thus, we relied on the Fuseki server with the version 0.2.3 which includes TDB 0.9.0 – a high performance RDF store. We used Fuseki with the default configuration. The RDFS reasoning is applied using an assembler description file.
BigData R is a high-performance RDF database which includes the NanoSparqlServer as SPARQL endpoint. We used the version 1.2.0 which comprises SPARQL UPDATE functionality. We employed the default setting except for setting up the RDFS reasoning within the RWStore.properties file.
Ontotext distributes three OWLIM editions. We deployed OWLIM-Lite 5.0.5 which builds on top of Sesame 2.6.5 and is freely available. Further, it is designed for datasets up to 100 million triples which fits our requirements.
Virtuoso provides an open source edition of their RDF store which includes a SPARQL endpoint. We installed version 6.1.5 und used the default setting. Inference is done only on runtime while querying.
Our benchmarks were conducted within a Ubuntu Linux 12.04 64bit virtual machine on a Intel Xeon CPU X5660 2.80GHz with 4 cores, 16GB RAM and 120GB virtual hard drive. The stores ran within Java 1.7.0 64bit runtime environment. If an application server was required, e. g., for OWLIM-Lite, we used Apache Tomcat 7.0.28. 2.2
Especially as we want to reuse existing semantic datasets within the Topic/S
project [
We defined 15 queries per dataset using the SPARQL 1.1 query language. To compare the performance between stores and datasets at once, we created six general queries, e. g., counting the number of distinct subjects or the occurrence of the properties. The further nine queries are dataset-dependent and designed for real-world scenarios in Topics, for instance to retrieve the names of persons Size (in MByte) Triples (in Mio) Instances (in k) Classes Properties
NY
Times
living in a dedicated time. Four of them are SELECTs with rising complexity,
e. g., by using UNION, regex filters, or subqueries. Query 11 is used to investigate
the DESCRIBE performance. In query 12 and 13 we especially considered the
performance of RDFS inference. The last two UPDATE queries delete and insert
triples. The complete list of queries can be found at [
In the following, we give a brief introduction to the metrics and the execution plan of our benchmark tests. The actual results of the tests are presented in Section 3.2. 3.1
In our benchmark we propose several metrics to capture different evaluation
aspects for the stores. They are tested with all four datasets which are different
in size and structure (c.f. Sect. 2.2). Furthermore, all the metrics are evaluated
with and without RDFS reasoning (except the multi-client performance tests
which were conducted exclusively with reasoning).
1. Loading time: At first, we measured the loading time of each dataset with
the stores three times and calculated the average.
2. Memory requirement : Further, we measured the memory consumption of
each store after the datasets were loaded.
3. Per-query type performance: Our main focus was to compare the query
performance of the stores. We mainly distinguish between the generic, the
dataset-specific, and the UPDATE queries. For our report we calculate the
average but also extract the min and max values.
4. Success rate: We also investigate the success rate of each query. We define
a query to be successful if it delivers the expected results without any error
or timeout.
5. Multi-client performance: For the multi-client scenario we measured the
average query performance as well as how many queries could be executed
within a 10 minutes time slot.
For the execution of the query benchmark we loaded the same dataset into
all installed and pre-configured stores. Further, we wrote a simple Java client
(test driver), which is available at our website [
From our benchmark we gain several insights we will discuss in the following
paragraphs. The selected Figures 1, 2 and 3 give a short summary of our results,
whereas our website [
Fig. 1 showcase the benchmark results with RFDS inference turned off. The first interesting finding is that OWLIM Lite performs best and Virtuoso worst to load all four datasets. BigData was fast but we identified a strange behaviour with the regex filters. The worked well for all datasets exempt from Jamendo. Here, all queries with a regex deliver no result. Further, query 3 and 13 on YAGO2 Core caused timeouts but BigDatas log didn’t provide any insight to solve the problem. Next, Virtuoso is the store of your choice if you had many UPDATE transactions (query 14 and 15) to handle. If we compare all INSERT and DELETE queries of all datasets, it is approximately 4 times faster then second-placed Fuseki. With regard to all queries made, the performance of OWLIM Lite and Virtuoso is nearly the same. Only with the 35 million dataset Virtuoso fell back. Finally, we identified that Fuseki scales really bad with subquery requests on NY Times and Jamendo. Unfortunately, we could not identify a particular reason as the query complexity is quite the same as for the other two datasets.
Our findings regarding the abilities of the stores to support RDFS reasoning are displayed in Fig. 2. Here, Fig. 2(a) confirms that OWLIM Lite is approximately 80% faster than Virtuoso regarding the load performance. As Fuseki is almost as fast like without reasoning, we run into performance issues with BigData. For our technical setup, it was not possible to load YAGO2 Core into the store because the created temporary file exceeded the disk space. But all in all, the memory consumption of the store changes only slightly (Fig. 2(b)). The average execution times of the queries (c.f. Fig. 2(c)) show that Virtuoso is still the fastest on UPDATEs by far – approximately 8 times faster than the secondplaced Fuseki. The performance of OWLIM and BigData decreases dramatically with this query type. Similar to the subquery problem of Fuseki we found that Virtuoso performs really bad with two generic queries (query 4 and 5) on the
Fuseki BigData OWLIM LITE Virtuoso (a) Loading (triples/second)
Ordiagtinaal Fuseki BigData OWLIM LITE Virtuoso (b) Memory requirements (in MByte) queries 1Ͳ 6
Virtuoso queries 7Ͳ 13
Fuseki
BigData
OWLIM LITE Fuseki
BigData OWLIM LITE Virtuoso Fuseki BigData OWLIM LITE (c) Average query execution time (in ms) per query type
Fuseki
BigData
OWLIM LITE Virtuoso
Original data
Virtuoso
Fuseki
BigData
OWLIM LITE Fuseki
BigData
OWLIM LITE 100 000 10000 1 000 100 10 1 Single Client 5 Clients 10 Clients FUSEKI
BigData
OWLIM Lite Jamendo dataset. As the reason is not obvious it strengthens our finding that benchmarks on real-world datasets is important. In the end, we must state that the query performance decreases in general.
The bar chart in Fig. 3 illustrates the average query execution time in our multi-client setup. OWLIM Lite scales best with factor 2 from single to five as well as 5 to 10 clients. Fuseki and BigData are running shoulder on shoulder as both scale quite linear to the number of clients. For Virtuoso, the performance of query 2 does not depend on the number of clients. But it unfolds issues for some queries, e. g., query 9 is around 533 time slower with 10 clients compared to the single client scenario.
In the end, we want to review the error rate in a qualitative way. First, we need to state that we faced issues regarding YAGO2 Core with RDFS inference turned on so that we did not measure any query performance. For instance, BigData produces a temporary file which exceeded the disk space of our virtual machine or Fuseki timed out for some of the queries. Second, our generic queries comprise the SPARQL COUNT statement so that we could easily compare the results. Within the RDFS inference scenario the benchmark was very surprising as for the queries 1, 2, 3, and 6 every triple store returned a different result. Further, OWLIM Lite was the only store which counts more triples for query 4 and 5. Thus, our urgent advice for your own project is to cross-check the results at random if you need to use RDFS reasoning. Third, another anomaly we faced was that BigData had problems with regex filters but only on Jamendo dataset. This underlines again the need to benchmark an RDF store with different realworld datasets. 4
In this paper we present yet another triple store benchmark as we did not find anyone with evaluation criteria like they are required for our research project: loading and querying real-world datasets, testing RDFS reasoning and SPARQL 1.1 queries, as well as conducting multiple queries in parallel. In the following, we discuss our four findings. As first result on our work with four up-to-date store we need to state, that comprehensive tests with real-world data are necessary. Otherwise it is not possible to detect anomalies like we identified. Second, every tested store allows for RDFS inference. But be careful as the result set may differ from store to store. Third, SPARQL 1.1 is well implemented nowadays. But the performance on UPDATE queries is varying. Here, Virtuoso stands out. Finally, we could not recommend any triple store in general as no store could win on all fields. Thus, the selection strongly depends on your specific project requirements. For our work, we will rely on OWLIM Lite because we need one which is fast in reading the datasets and multi-client query processing.
As we had problems with YAGO2 Core and inference, especially with regard to our technical setup, we are evaluating the requirements for bigger datasets. Besides the core version we like to benchmark the stores with YAGO2 Full as well. Another future work is to evaluate the performance of OWL reasoning for some common constructs, e. g., cardinalities. Therefore, we need to identify suitable real-world datasets. 5
Work on this paper is partly funded within the Topic/S project by the European Social Fund / Free State of Saxony, contract no. 99457/2677.