The Linked Data paradigm which is now the prominent enabler for sharing huge volumes of data by means of Semantic Web technologies, has created novel challenges for non-relational data management technologies such as RDF and graph database systems. Benchmarking, which is an important factor in the development of research on RDF and graph data management technologies, must address these challenges. In this paper we present the Semantic Publishing Benchmark (SPB ) developed in the context of the Linked Data Benchmark Council (LDBC) EU project. It is based on the scenario of the BBC media organisation which makes heavy use of Linked Data Technologies such as RDF and SPARQL. In SPB a large number of aggregation agents provide the heavy query workload, while at the same time a steady stream of editorial agents execute a number of update operations. In this paper we describe the benchmark's schema, data generator, workload and report the results of experiments conducted using SPB for the Virtuoso and GraphDB RDF engines.
Non-relational data management is emerging as a critical need in the era of a new data economy where heterogeneous, schema-less, and complexly structured data from a number of domains are published in RDF. In this new environment where the Linked Data paradigm is now the prominent enabler for sharing huge volumes of data, several data management challenges are present and which RDF and graph database technologies are called to tackle. In this context, benchmarking is an important factor in the development of research of the aforementioned technologies that are called to address these new challenges.
To test the performance of SPARQL query evaluation techniques, a
number of benchmarks have been proposed in the literature [
Existing RDF benchmarks do not fully cover important RDF and SPARQL
intricacies since data remains relational at heart, the workload is generally
formed by simple read-only query operations, and the queries have not been
specifically designed to stress the systems and in particular their debilities. In
addition, existing benchmarks do not consider real use case scenarios (except
DBSB [
In this paper we present the Semantic Publishing Benchmark (SPB ) developed in the context of the Linked Data Benchmark Council (LDBC)4 European project. It is inspired by the Media/Publishing industry, and in particular by BBC’s5 “Dynamic Semantic Publishing” (DSP) concept. BBC maintains sets of RDF descriptions of its catalogue of creative works such as articles, photos, and videos.
SPB is designed to reflect a scenario where a large number of aggregation
agents provide the heavy query workload, while at the same time a steady stream
of editorial agents implement update operations that insert and delete creative
works. Journalists use the aggregation agents to query existing creative works
and use the retrieved data in order to create new ones using the editorial agents.
The Semantic Publishing Benchmark includes:
– a data generator that uses ontologies and reference datasets provided by
BBC, to produce sets of creative works. The data generator supports the
creation of arbitrarily large RDF datasets in the order of billions of triples
that mimic the characteristics of the reference BBC datasets.
– the workload in SPB is defined by the simultaneous execution of editorial
and aggregation agents, simulating a constant load generated by end-users,
journalists, editors or automated engines. The workload is designed to reflect
a scenario where a large number of aggregation agents provide the heavy
query workload, while at the same time a steady stream of editorial agents
implement update operations that insert and delete creative works.
The SPB queries are defined in a way that tackle the choke points [
The paper is structured as follows: Section 2 discusses briefly SPB , Section 3
discusses the experiments we conducted running SPB for the Virtuoso [
The scenario of SPB is based around a media organization that maintains RDF descriptions of its catalogue of creative works or journalistic assets (articles, photos, videos), papers, books, movies among others. Creative works are valid instances of BBC ontologies that define numerous concepts and properties employed to describe this content.
In this section we discuss the ontologies and reference datasets (Section 2.1) used by SPB , its data generator (Section 2.2) that uses the ontologies and reference datasets provided by BBC to produce a set of creative works, the queries that the benchmark introduces (Section 2.3) and finally the employed performance metrics (Section 2.4). 2.1
SPB uses seven core and three domain RDF ontologies provided by BBC. The former define the main entities and their properties, required to describe essential concepts of the benchmark namely, creative works, persons, documents, BBC products (news, music, sport, education, blogs), annotations (tags), provenance of resources and content management system information. The latter are used to express concepts from a domain of interest such as football, politics, entertainment among others.
The ontologies are relatively simple: they contain few classes (74), 29 and
88 data type and object properties respectively, and shallow class and property
hierarchies (65 rdfs:subClassOf, 17 rdfs:subPropertyOf). More specifically,
the class hierarchy has a maximum depth of 3 whereas the property
hierarchy has a depth of 1. They also contain restrictions (107 rdfs:domain and 117
rdfs:range RDFS properties). The BBC ontologies also ontain 8 owl:oneOf class
axioms that allow one to define a class by enumeration of its instances and 2
owl:TransitiveProperty properties. Additionally the simplest possible flavor
of OWL has been used (owl:TransitiveProperty, owl:sameAs) for nesting of
geographic locations or relations between entities in the reference dataset. A
detailed presentation of the ontologies employed by SPB can be found in [
SPB also uses reference datasets that are employed by the data generator to produce the data of interest. These datasets are snapshots of the real datasets provided by BBC; in addition, a GeoNames and DBPedia reference dataset has been included for further enriching the annotations with geo-locations to enable the formulation of geo-spatial queries, and person data. 2.2
The SPB data generator produces RDF descriptions of creative works that are valid instances of the BBC ontologies presented previously. A creative work is described by a number of data value and object value properties; a creative work also has properties that link it to resources defined in reference datasets: those are the about and mentions properties, and their values can be any resource. One of the purposes of the data generator is to produce synthetic large (in the order of billions of triples) datasets in order to check the ability of the engines to scale. The generator models three types of relations in the data: Clustering of data The clustering effect is produced by generating creative works about a single entity from reference datasets and for a fixed period of time. The number of creative works starts with a high peak at the beginning of the clustering period and follows a smooth decay towards its end. The data generator produces major and minor clusterings with sizes (i.e., number of creative works) of different magnitude. By default five major and one hundred minor clusterings of data are produced for one year period. Example of clusterings of data could be news items that are about events starting with a high number of journalistic assets related to them and following a decay in time as they reach the end of time period, a tendency that mirrors a real world scenario in which a ’fresh’ event is popular and its popularity decreases as time goes by.
Correlations of entities This correlation effect is produced by generating creative works about two or three entities from reference data in a fixed period of time. Each of the entities is tagged by creative works solely at the beginning and end of the correlation period and in the middle of it, both are used as tags for the same creative work. By default fifty correlations between entities are modelled for one year period. Such an example of data correlation could be that several ’popular’ persons (e.g., Stross-Cahn and Barroso) are mentioned together by creative works for a certain period of time.
Random tagging of entities Random data distributions are defined with a bias towards popular entities created when the tagging is performed, that is when values are assigned to about and mentions creative work properties. This is achieved by randomly selecting a 5% of all the resources from reference data and mark them as popular when the remaining ones are marked as regular. When creating creative works, 30% percent of them are tagged with randomly selected popular resources and the remaining 70% are linked to the regular ones. Example for random taggings could be every-day events which become less important several days after their start date. Distributions of about and mentions tags analysed from a ’live’ dataset provided by the BBC are reproduced in generated data. Table 1 shows the distribution of total about and mentions tags found in creative works and also shows their individual distributions.
Additional modification of current version has been added which triples the number of about and mentions tags used in Creative Works, thus providing a better interconnectedness of entities across whole dataset. This random generation of data concerns only one third of all generated data; the remaining data is generated with correlations and clustering effects modeled as previously described.
The SPB data generator operates in a sequence of phases: 1. ontologies and reference datasets are loaded in an RDF repository; 2. all instances of the domain ontologies that exist in the reference datasets are retrieved by means of predefined SPARQL queries that will be used as values for the about and mentions properties of creative works; Amount 3. from the previous set of instances, the popular and regular entities are selected; 4. the generator produces the creative works according to the three properties discussed previously. SPB is designed to reflect a scenario where a large number of aggregation agents provide the heavy query workload requesting creative works, while at the same time a steady stream of editorial agents implement update operations that insert and delete creative works.
Aggregation Queries SPB queries are valuable in the sense that they stress
important technical functionalities that systems must tackle. P. Boncz [
SPB queries are defined in a way that tackle the choke points that each RDF store needs to address in order to satisfy the requirements raised from real world use cases. The benchmark comes with two types of workloads: the base and advanced versions. The former tackles challenges related to SPARQL query processing, whereas the latter extends the former by including versioning and backup database functionalities. In this paper we restrict ourselves to the presentation of the base version of SPB benchmark since it addresses features interesting for SPARQL query processing. A subset of SPB queries are based on real ones obtained from the applications that BBC journalists use to access existing creative works in order to create new ones; the remaining queries are based on these complex queries and are enhanced with different features of SPARQL 1.1.
We first present the choke points (CP) SPB queries implement and then we provide a detailed explanation of the SPB queries and the choke points they implement. cp1: join ordering This choke point tests if the optimizer can evaluate the trade-offs between the time spent to find the best execution plan and the quality of the output plan. It also tests the ability of the engine to consider cardinality constraints expressed by the different kinds of properties defined in the ontologies (such as functional properties and inverse functional properties). cp2: aggregation Aggregations are implemented with the use of sub-selects in the SPARQL query; the optimizer should recognize the operations included in the sub-selects and evaluate them first. cp3: optional and nested optional clauses This choke point tests the ability of the optimizer to produce a plan where the execution of the optional triple patterns is the last to be performed since optional clauses do not reduce the size of intermediate results. cp4: reasoning Reasoning tests are used to check whether the systems handle efficiently RDFS and OWL constructs. cp5: parallel execution of unions This choke point tests the ability of the optimizer to produce plans where unions are executed in parallel. This is especially helpful if the involved subqueries produce a large number of results. cp6: optionals with filters This specific choke point tests the ability of the engines to execute as early as possible such expressions in order to eliminate a possibly large number of intermediate results. Note that this choke point is similar to cp3 with the difference that here the expression contains filters that bind variables to specific values. Filters reduce (in some cases significantly) the size of the intermediate results. Such queries (related to CP3 and CP6) are of high importance as they often found in real world scenarios. cp7: ordering This test checks the ability of the optimizer to choose query plan(s) that facilitate the ordering of results. cp8: geo-spatial predicates This choke point tests the ability of the system to handle queries for geospatial data; queries that mention entities within a special geospatial range address this technical challenge. cp9: full text search Queries that involve the evaluation of regular expressions on data value properties of resources address this choke point. cp10: duplicate elimination This choke point tests the ability of the system to identify duplicate entries and eliminate them during the creation of intermediate results. cp11: complex filter conditions Filter conditions that involve negation, conjunction and disjunction can be used to test if the optimizer is able to split the filter conditions into conjunctions of conditions and execute them in parallel and as soon as possible.
In Table 2 we provide the 12 queries that comprise the base version of SPB . For each of them we provide a text description and we enumerate the choke points it addresses. All the SPARQL queries can be found online6. Queries Q1, Q2, Q3, Q4, Q8 use the CONSTRUCT, and Q5, Q6, Q7, Q9, Q10, Q11 and Q12 use the SELECT SPARQL modifiers.
Query Description Choke Points Retrieve the 10 most recent creative works, that are about or mention cp1, different topics. For each creative work a graph is returned that com- cp2, prises of the work’s title, shortTitle, dateCreated, dateModified, descrip- cp3, tion, primaryFormat and primaryContentOf properties; if the creative cp4, work has a thumbnail, then return its thumbnailAltText and thumb- cp5 nailType. Also retrieve existing properties of the topics that the creative work is about or mentions: aboutLabel, aboutShortLabel, aboutPreferredLabel, mentionsLabel, mentionsShortLabel and mentionsPreferredLabel. Retrieve details about a given resource that is an instance of class cp1, Creative Work and its subclasses, namely its title, dateCreated, date- cp3, Modified, the topic the resource is about, its primaryContentOf and the cp4 webDocumentType thereof.
Retrieve a list of creative works that are instances of classes BlogPost or cp3, NewsItem, that are about a specific topic, the value of primaryFormat cp5, is one of TextualFormat, InteractiveFormat or PictureGalleryFormat. cp6, If the creative work has an audience then the creative work should be cp10, obtained using this value. Given the retrieved list of creative works, cp11 return for each resource its related nodes, and then order the result in descending order of the value of their creationDate property.
Return a list of all creative works with all their properties, that are cp1, about a given topic, have a given primaryFormat, and are instances of cp7 a certain subclass of class CreativeWork. The results should be ordered by property creationDate and only N results should be returned.
Return the list of most popular topics that creative works of a given cp1, type are about, have a given audience, and they have been created in a cp2, specific time range (dateModified is between a start and an end date). cp3, For each retrieved topic get its properties (if they exist) cannonical- cp7, Name and preferredLabel; and if any of the labels exist, return it as a cp11 result along with the count of topics with this label. The result should be returned on descending order of the count of labels.
Retrieve all instances of class CreativeWork that mention a geo-location cp7, that is within the boundaries of a given rectangle area. Along with cp8, each retrieved creative work retrieve property geonamesId and its lat cp10, (latitude) and long (longitude) values. Limit the result to 100 creative cp11 works (geo-spatial query).
Retrieve properties dateModif, title, category, liveCoverage, audience for cp1, all creative works that are of a given type. The value of property date- cp7, Modif of the retrieved creative works should be within a certain time- cp11 range. Return 100 results ordered in ascending order by their dateModif. Q1 Q2 Q3 Q4 Q5 Q6 Q7 6https://github.com/ldbc/ldbc_spb_bm_2.0/tree/master/datasets_and_queries /sparql/basic/aggregation_standard
Retrieve the graphs of resources that are instances of class Creative- cp3, Work considering also its subclasses that comprise the resources’ type, cp4, title, description, dateCreated, dateModified, category, the topic they cp9, are about, their primaryContentOf and the latter’s webDocumentType. cp11 Each of the returned resources should contain in its title or description a given string. (full text query) Retrieve 10 similar creative works, by calculating their similarity score. cp2, The creative works should be ordered by the computed score and in cp7, descending order of the value of property dateModified. cp10 Retrieve creative works that mention locations in the same province cp1, (A.ADM1) as the specified one. Additional constraint on time interval cp4, further limits returned results. cp5, cp11 Retrieve a list of the most recent Creative Works that have tagged cp1, with entities, related to a specific popular entity from reference dataset. cp4, Relations can be (inbound and outbound; explicit or inferred) cp5, cp7 Retrieve the descriptions of the latest creative works tagged with a cp1, specific location. Consider that the description of each specific Cre- cp2, ativeWork is stored in dedicated named graph. The result should in- cp7, clude only the explicit statements about the creative work, without cp10 owl:sameAs equivalence and without statements inferred otherwise .
Table 3 shows the SPARQL features that each of the queries in the SPB query mix implement. By complex filters we refer to filters that contain conjunction and disjunction of predicates, and by negation we refer to the use of negation associated with the bound SPARQL operator employed in the filters. Query Q1 is a very complex query that contains 11 optionals with 4 of them having nested optional clauses. Query Q3 contains the majority of the SPARQL features (except aggregates, group by, and regular expressions in filters).
Aggregation agents simulate the retrieval operations performed by journalists, end-users or automated search engines by executing a mix of aggregation queries of type: aggregation, search, statistical, full-text, geo-spatial, analytical, drill-down and faceted search. Each aggregation agent will execute a mix of those query types in a constant loop, until the benchmark run finishes. Each agent executes a query and waits for response (or a time-out), after receiving the response next query is executed (queries executed by agents are not of the same type). Query order of execution is pseudo-randomly chosen following an even distribution for each query defined in the benchmark’s configuration. Update Queries Editorial agents simulate the editorial work performed by journalists, editors or automated text annotation engines by executing the following update queries: Feature Q1 optionals 7 nested optionals 4 union order by 1 distinct 1 limit 1 nested queries 1 group by aggregates negation complex filters regexp
Q2 Q3 4 1 - - 3 - 1 - 1 - 1 - 1 - - - 1 - 2 - – Insert operations generate new creative work descriptions (content metadata) following the models and distribution rules defined in Section 2.2. Each creative work is added to the database in a single transaction by execution of an insert SPARQL query. – Update operations update an existing creative work. Update operation consists of two actions, executed in one transaction, following the BBC’s use-case for update of a creative work. First action is to delete the context where a creative work description resides along with all its content. Second is to insert the same creative work (using its current ID) with all its properties - current and updated ones. – Delete operations delete an existing creative work. Delete operation will erase the context where a creative work resides along with all of its content.
Each editorial agent will execute a mix of editorial operations in a constant loop, until the benchmark run has finished. Editorial operations executed by an agent are chosen pseudo-randomly following the distribution: 80% INSERT operations, 10% UPDATE operations, 10% DELETE operations. Such distribution follow a similar to live datasets pattern where a massive amount of new data being added to the database (inserts and updates) and a minor amount of data being deleted from it. Such rates are open and freely configurable, thus each audited run should include a statement about the exact distribution values. 2.4
Performance metrics produced by the SPB benchmark describe how fast an RDF database can execute queries (by simultaneously running aggregation agents) while at the same time executing editorial operations (by simultaneously running editorial agents) and operating over an amount of generated data in the RDF database. The data generator of SPB is capable of producing data in different sizes (up to billions of triples), thus allowing to have performance results for various scales. SPB outputs two types of performance metrics: 1. Minimum, Maximum and Average execution times for each individual query and editorial operation during the whole benchmark run. 2. Average execution rate per second for all queries and editorial operations.
Performance metrics are produced per-second during the benchmark run and are saved to log files. Further all of the information related to query execution and query results is saved in log files with different level of details. Regarding the performance metric (1), due to space restriction, we only present in the experimental evaluation the average execution times for the SPB workload, but all the results are available online7. 2.5
SPB comes with test and benchmark drivers both distributed as a jar file along with the BBC ontologies and reference datasets, the queries and updates discussed earlier and the configuration parameters for running the benchmark and for generating the data. The data generator uses configuration files that must be edited appropriately to set the values regarding the dataset size to produce, the number of aggregation and editorial agents, the query timeout etc. The distributions used by the data generator to produce the datasets could also be edited. The benchmark, that is available on GitHub8, is very simple to run (once the RDF repository used to store the ontologies and the reference datasets is setup, and the configuration files updated appropriately) from command line.
The test driver generates substitution parameters for the SPB queries during data generation. The use of these parameters ensures that the benchmark is deterministic, i.e., different runs of the benchmark will employ the same values, and hence produce comparable results. During one benchmark run the driver goes through an initialization phase where one instance per benchmark query is created by selecting parameters from the respective query parameter file. Once each query executes the last set of parameters, the next time it is executed (and for the same run), it reiterates from the first set of parameters in the list.
The benchmark driver allows one to specify the size of the sets of substitution parameters per query, which are saved in files along with the generated data. The benchmark produces three kinds of files that contain (a) brief information about each executed query, the size of the returned result, and the execution time (b) the detailed log of each executed query and its result and (c) the benchmarking results. 3
In this section we report the results we obtained when running SPB for the RDF
engines Virtuoso [
threads) with 192GB of RAM and running CentOS release 6.2 x86_64. The system has 6 2TB SATA-2 and 2 512GB SATA-3 disks under Patsburg 6-Port SATA AHCI Controller.
We used GraphDB, a semantic graph database management system, Version 6.2 Enterprise Edition. The experiments conducted for GraphDB, run on a single Intelr Xeonr CPU E5-1650 v3 at 3.5GHz (a total of 6 cores/12 threads) with 64GB of main memory and running Ubuntu 14.04 x86_64. The system has 1 500GB SATA-3 in 7200RPM for the operating system only and 2 400GB SSDs, Samsung 845DC PRO.
Our aim is not to make a detailed comparison of the performance of the different systems but provide an idea of how they perform regarding the SPB benchmark and this is why we did not use the same configuration for Virtuoso and GraphDB. A complete description of the experiments reported here can also be found online (see Section 2.5).
Datasets: For our experiments we produced three datasets of different scales that correspond to 64M (SF1), 256M (SF3) and 1B (SF5) triples. GraphDB has decided to publish results for SF1 (64M) and SF3 (256M) only based on the most common use-cases in the industry where commodity hardware is widely adopted. Table 4 shows the statistics of the datasets produced by the SPB data generator that we will use for our experiments. We provide the number of explicit (i.e., generated) and total triples for each dataset (implicit and explicit), as well as the number of creative works since these are the resources requested and updated by the SPB workload. GraphDB supports forward chaining and computes the closure of the dataset during data generation, consequently we report the implicit triples. The slight deviation in numbers comes from the configuration of the data generator e.g. how many threads have been configured to generate the creative works. Note that the data generator produces creative works, each resource associated with 25 to 29 triples. So threads will stop producing creative works once the dataset size as specified in configuration file has been reached. Nevertheless, if some thread is in the middle of generating a creative work, it will generate that last resource beforing ending.
#explicit #implicit
SF1 63.709.210 N/A Virtuoso SF3 255.659.835 N/A
SF5 999.609.164 N/A
#total #cworks 63.709.210 1.489.675 255.659.835 8.821.474 999.609.164 37.233.029
GraphDB SSFF13 26546..000000..000000 26296..121420..833461 148332..121420..833461 18..488291..730910 Configuration: For Virtuoso we measured the performance results on an enterprise grade hardware and SPB driver configuration of 22r/2w threads, while GraphDB has chosen to use a modest system configuration and an SPB driver configuration of 8r/2w threads (see Table 5 for the configuration details). Both vendors were free to choose their optimal configuration of the reading and writing threads (aggregation and editorial agents) of the SPB driver as well as to select the system configuration which will be used for running the benchmark in order to achieve the best performance results.
Virtuoso GraphDB Scale Factor SF1 SF3 SF5 SF1 SF3 Aggregation agents 22 22 22 8 8 Editorial agents 2 2 2 2 2 Data generator workers 4 4 4 8 8 Warm up period (min) 10 10 5 10 10 Duration (min) 20 30 20 20 30
Timout (min) 5 5 5 5 5 Workload: In our experiments we used the Basic Version of SPB that consists of the 12 queries discussed in Section 2.3. Recall that SPB queries are expressed in SPARQL 1.1. In the case of Virtuoso that employs backward reasoning and for queries that involved reasoning we used the option transitive and enabled rdfs_rule_set option that implements the traversal of RDFS hierarchies along the rdfs:subClassOf and rdfs:subPropertyOf relations. Data Generation and Loading Times: The time needed to load the ontologies and reference datasets in both systems is in the order of few milliseconds and we do not report it here. The dataset generation and loading times, are shown in Table 6. Data generation is slower on Virtuoso’s hardware because Virtuoso’s server uses HDD drive while hardware configuration for GraphDB uses SSD drive and the data generation process is intensive in terms of I/O operations to the storage device Regarding the loading times, GraphDB loads data much slowly (in the case of SF3 the difference is up to one order of magnitude) because of the forward chaining reasoner which materializes each inferred fact when adding new ones to the database, something not done by Virtuoso.
Performance Metrics: Table 7 shows the average execution rates per second and Table 8 provides the detailed execution times for each of the 12 benchmark queries. These results are provided for both engines and for all of the scale factors (SF1, SF3 and SF5). For SF1, Virtuoso executes 150 queries per second on 22 aggregation agents (reading threads), whereas GraphDB achieves about 100 queries per second using 8 reading threads.
A large difference in the update rate exists for the two engines. Recall that, as described in Section 2.3, the editorial workload consists of 80% of inserting
SF1 Virtuoso SF3
SF5 GraphDB SSFF13 GraphDB SSFF13 new facts, 10% of updating and 10% of deleting of existing ones. GraphDB’s low update rate is due to the fact that it uses a forward-chaining materialization of inferred statements which adds an overhead when inserting new data.
Virtuoso uses backward-chaining to compute the necessary implicit triples during query execution; this is performed in memory and inserting a new fact in the database does not trigger any materialization of additional knowledge. Consequently, the update rate is higher. This approach saves time during insertion of new data, but could be tricky during query execution: all queries that require materialization Q1-Q8 and Q11 are slower in Virtuoso than for GraphDB, as such materialization took place during query execution.
Further increasing the scale factor of generated data from SF1 to SF3, which effectively consists of 239M generated statements and 25M statements of reference data, shows a drop in query performance for both GraphDB (60%) and Virtuoso (45%) which is expected because of the four times larger amount of data that the SPB benchmark operates on.
In the reported experiments, Virtuoso stores RDF triples in a single RDF
quad table with Virtuoso’s default index configuration and considers hash join
as a possible join type. Considering a hash join as a possibility always slows down
compilation, and improves some queries and slows others down, not greatly
affecting the average throughput. We run additional (unaudited) experiments with
Virtuoso where we introduced query plan caching that raises the Virtuoso
score from 80 queries/per second (qps) to 144 qps for SF3. The rationale for
considering hash join in the first place is that analytical workloads such as SPB ’s
heavily rely on this. A good TPC-H score is simply unfeasible without this9 and
hash join is indispensible if RDF is to be a serious contender beyond serving
lookups. The decision for using this however depends on accurate cardinality
estimates on either side of the join. Previous work [
In this Section we are going to provide a short overview of the existing RDF benchmarks, and an analysis based on the following dimensions: the datasets (including data generators in the case of synthetic benchmarks) and query workloads. In our presentation we distinguish between benchmarks that use real datasets and those that produce synthetic datasets using special purpose data generators.
DBPedia10, UniProt KnowledgeBase (UniProtKB) [
SPB uses real reference datasets from the BBC to produce synthetic datasets of arbitrary size (up to billions of triples) that mimic the characteristics of the former. The workload comprises of both real and synthetic queries and updates that consider all SPARQL 1.0 operators that are more complex than the workloads proposed in the previously discussed benchmarks.
In addition to benchmarks using real-world datasets synthetic RDF
benchmarks have been proposed in the literature. The Lehigh University Benchmark
(LUBM) [
SPB goes one step beyond the aforementioned benchmarks given that the generator can produce arbitrarily large datasets (in the order of billions of triples), proposes a much more complex workload than the previous ones consisting of queries that contain a fairly large number of triple patterns considering all SPARQL 1.0 operators as well as nested queries of high complexity. In this paper we presented the Semantic Publishing Benchmark (SPB ) developed in the context of the Linked Data Benchmark Council (LDBC) European project inspired by BBC’s Dynamic Semantic Publishing” (DSP) concept. We presented in detail the benchmark’s schema, data generator, query workload and performance metrics, and reported on the results of a set of experiments we conducted against the Virtuoso and GraphDB RDF engines. In the next versions of SPB we plan to use more expressive OWL ontologies for representing information. We also plan to improve the modelling of content-to-entity cardinalities as to enrich query sets and realistic query frequencies by taking advantage of FT data and query logs respectively.
The work presented in this paper was funded by the FP7 project LDBC (#317548)