We have compared the performance of five non-commercial triple stores, Virtuoso-open source, Jena SDB, Jena TDB, SWIFT-OWLIM and 4Store. We examined three performance aspects: the query execution time, scalability and run-to-run reproducibility. The queries we chose addressed different ontological or biological topics, and we obtained evidence that individual store performance was quite query specific. We identified three groups of queries displaying similar behavior across the different stores: 1) relatively short response time, 2) moderate response time and 3) relatively long response time. OWLIM proved to be a winner in the first group, 4Store in the second and Virtuoso in the third. Our benchmarking showed Virtuoso to be a very balanced performer - its response time was better than average for all the 24 queries; it showed a very good scalability and a reasonable run-to-run reproducibility.
1 Introduction
Semantic Web technologies are increasingly being adopted by the scientific
community, and Life Sciences researchers are no exception [
Currently, there are several triple store solutions [
The development of triple stores has flourished during the last 5 years. Currently,
there are more than 20 systems available [
Much of the benchmarking done previously on triple stores was based on artificial
data or a set of triples that could at best only mimic a realistic ontology. Among the
“standard” sets used are: the Lehigh University Benchmark (LUBM [
Here, we present “the NTNU benchmark”, which is the work we undertook using
five popular triple store implementations, and report the outcome of this
benchmarking. In comparison with previous benchmarkings [
The set of triple store implementations included Virtuoso OpenSource 6.0.0, Swift OWLIM 2.9.1, 4Store 1.0.2, Jena SDB 1.3.1, Jena TDB 0.8.2. The stores were run under Centos 5 operating system. The details of software configuration are available on request.
The benchmarking was performed on a Dell R900 machine with 24 Intel(R) Xeon(R) CPUs (2.66GHz). The machine was equipped with 132G main memory and 14x500GB 15K SAS hard drives.
The ten graphs constituting the Cell Cycle Ontology (CCO) [
The most salient features of the queries used for our benchmarking are summarized in Table 1.
C@8@?! =FGHF!IJ! G@.?@AK?! FHLHM! 09D/E/36 09D/E/36 09D/E/36 9136;596 $
K=NA?! 9136;596 $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ ! "# "% "& "' "( ") "* "+ ", "#"## "#% "#& "#' "#( "#) "#* "#+ "#, "%"%# "%% "%& "%' $ $ $ $ $ $
As can be seen from Table 1, this collection of queries encompasses a broad range of features and combinations thereof. This ensures a comprehensive assessment of the performance of the triple stores.
In order to get a bird's eye view on the performance of the stores we aggregated the
response times into the single cumulative total response time and estimated the
average relative standard errors for each of the stores (Table 2). (Please note that
OWLIM does not support the COUNT operator, therefore the values for this store do
not include data for queries Q17, Q19, Q20. Q21).
The total execution time varied over a very broad range and some of the stores (most
notably Jena TDB and OWLIM) displayed an unexpectedly high run-to-run
variability. On the basis of these data Virtuoso emerges as an overall winner, with by
far the best total execution time and a relatively small run-to-run variation. However,
the picture changes radically when we look into the query- specific behavior (Table
3).
The table makes clear that all the stores behave in a query specific manner. A
highly query-specific behavior has been also observed by Bizer and Schultz [
% $&'
$ 0 #&' )3+A # ?-@ "&'
" !&' ! queries. The list of features shared by the queries Q3 and Q18 includes simple filters, more than 8 triple patterns and a REGEX operator. At present it is not possible to determine which of these features or a combination thereof are responsible for the long execution time.
Finally, we wanted to compare the stores with respect to their scalability. The averaged response times were summed over all the queries (except for the queries Q17, Q19, Q20. Q21 for OWLIM) and plotted against the total number of triples in the graphs (Figure 1).
! "!!!!!! #!!!!!! =3+*)>?0.
$!!!!!! %!!!!!!
As can be seen from the figure OWLIM scales up extremely well, with Virtuoso and Jena SDB as second best. 4Store demonstrated the poorest performance with respect to scalability. However, as pointed out earlier, the behavior of OWLIM and 4Store is strongly affected by a few outliers. Therefore, to eliminate the impact of the outliers we excluded the three slowest queries Q3, Q14 and Q18 from the plot (Figure 2).
"&( "&' "&% "&# 1 " *4,B!&( @.A !!&&'% !&# ! $!!!!!! %!!!!!!
Although the mutual arrangement of the individual graphs on the plot changed in
favor of OWLIM and 4Store, the conclusion drawn previously about the scalability
did not change.
4 Conclusions
We have compared the performance of five popular triple stores, Virtuoso-open
source, Jena SDB, Jena TDB, Swift OWLIM and 4Store, in three aspects – the query
execution time, scalability and run-to-run reproducibility. According to our results
there is no absolute winner within this set of stores. Instead, the performance seems to
be quite query-specific. Nevertheless, it was possible to identify three groups of
queries displaying similar behavior with respect to the different stores: 1) relatively
short response time, 2) moderate response time and 3) relatively long response time.
OWLIM proved to be a winner in the first group, 4store in the second and Virtuoso in
the third. Virtuoso emerged from our benchmarking as a very balanced performer –
its response time was better than average for all the 24 queries; it showed a very good
scalability and a reasonable run-to-run reproducibility. Even though in our study we
used only moderately large triple stores, others demonstrated that Virtuoso excels
when confronted with much larger stores, up to 100-200 M triples [
Data availability. The rdf files used for uploading the triple stores are downloadable
from the CCO web site [
Acknowledgments. The authors appreciate the assistance of E. Jensen with setting up the triple stores used in this benchmarking. V. Mironov was supported by FUGE Midt Norge.