ParlBench is a scalable RDF benchmark modelling a large scale electronic publishing scenario. The benchmark o ers large collections of the Dutch parliamentary proceedings together with information about members of the parliament and political parties. The data is real, but free of intellectual property rights issues. On top of the benchmark data sets, several realistic application benchmarks as well as targeted micro benchmarks can be developed. This paper describes the benchmark data sets and 28 analytical queries covering a wide range of SPARQL constructs. The potential use of ParlBench is demonstrated by executing the query set for 8 di erent scalings of the benchmark data sets on Virtuoso RDF store. Measured on a standard laptop, data loading times varied from 43 seconds (for 1% of the data set) to 48 minutes (for the complete data set), and execution of the complete set of queries (1520 queries in total) varied from 9 minutes to 13 hours.
RDF stores are the backbones of RDF data driven applications. There is a wide range of RDF stores systems available1 together with various benchmark systems2 to assess performances of the systems.
As discussed in the Benchmark Handbook [
With this work we propose the ParlBench application benchmark that closely mimics a real-life scenario: large scale electronic publishing with OLAP-type queries. ParlBench consists of (1) real life data and (2) a set of analytical queries developed on top of these data. ? This work was supported by the EU- FP7 (FP7/2007-2013) project ENVRI (grant number 283465).
The benchmark data sets include the Dutch parliamentary proceedings, political parties and politicians. The ParlBench data t very well the desiderata of Gerhard Weikum's recent Sigmod blog3: it is open, big, real, useful, linked to other data sources, mixing data-values and free text, and comes with a number of real-life workloads.
The queries in the benchmark can be viewed as coming from one of two use cases:
create a report or perform a scienti c research. As an example of the latter, consider the
question whether the performance of males and females di ers in parliament, and how
that has changed over the years. To enable more comprehensive analysis of the RDF
stores' performances, we grouped the benchmark queries into four micro benchmarks
[
The paper is organized as follows. Section 2 gives an overview of the related work. Section 3 describes the benchmark data sets. In Section 4 we de ne the benchmark queries and present the micro benchmarks. The evaluation of the ParlBench benchmark on the Virtuoso RDF store is discussed in Section 5. 2
There is a number of RDF store benchmarks available. The most relevant benchmarks
to our work are discussed further. The Berlin SPARQL Benchmark (BSBM) [
The SPARQL Performance Benchmark (SP2Bench) [
Both the Berlin and SP2Bench use synthetically generated data sets, whereas, the
DBpedia SPARQL Benchmark (DBPSB) [
The benchmark consists of ve conceptually separate data sets summarized in Table 1: Members : describes political players of the Dutch parliament.
Parties : describes Dutch political parties. 3 http://wp.sigmod.org/?p=786 Proceedings : describes the structure of the Dutch parliamentary proceedings. Paragraphs : contains triples linking paragraphs to their content.
Tagged entities : contains triples linking paragraphs to DBpedia entities indicating that these entities were discussed in the paragraphs. dataset # of triples size # of les members 33,885 14M 3,583 parties 510 612K 151 proceedings 36,503,688 4.15G 51,233 paragraphs 11,250,295 5.77G 51,233 tagged entities 34,449,033 2.57G 34,755
TOTAL: 82,237,411 13G 140,955
The data model of the benchmark data sets is described in Appendix A. 3.1
The size of the ParlBench data sets can be changed in di erent ways. The data set can be scaled by the number of included proceedings. All proceeding les are ordered chronologically. The scaled data set of size 1=n consists of every n-th le in this list, plus the complete Parties and Members sets. Optionally, one can include Tagged entities and/or Paragraphs data sets to the test collection. In this case Paragraphs and Tagged Entities are scaled accordingly to the included proceedings, i.e., only paragraphs and/or tags pointing to id's in the chosen proceedings are included. 4
ParlBench provides 19 SPARQL queries. The queries were grouped into four micro benchmarks: Average: 3 queries, numbered from A0 to A2, retrieve aggregated information. Count: 5 queries, numbered from C0 to C4, count entities that satisfy certain conditions.
Factual: 6 queries, numbered from F0 to F5, retrieve instances of a particular class that satisfy certain conditions.
Top 10: 5 queries, numbered from T0 to T4, retrieve the top 10 instances of a particular class that satisfy certain ltering conditions.
All the queries are listed in Appendix B. Their SPARQL representations can be seen in Appendix C. The benchmark queries cover a wide range of the SPARQL language constructs. Table 1 shows the usage of SPARQL features by individual query and distribution of the features across micro benchmarks. 5
In this section we demonstrate the application of our benchmark on the OpenLink
Virtuoso RDF native store (OSE)4. Tested on the Berlin benchmark, Virtuoso showed
one of the best performance results among other systems [
Loading Time The loading time is the time for loading RDF data into an RDF store. The benchmark data sets are in RDF/XML format. The time is measured in seconds. Loading of data into Virtuoso was done one data set at a time. For the loading of Parties and Members we used the Virtuoso RDF bulk load procedure. For Proceedings we used the Virtuoso function DB.DBA.RDF LOAD RDFXML MT to load large RDF/XML text.
Query Response Time The query response time is the time it takes to execute a SPARQL query. To run the queries programmatically, we used isql, the Virtuoso interactive SQL utility. The execution time of a single query was taken as the real time returned by the bash /usr/bin/time command. 10 permutations of the benchmark queries were created, each containing 19 SPARQL queries.
Before starting measuring the query response time, we warmed-up Virtuoso by running 5 times 10 di erent permutations of all 19 queries of the benchmark. In total, 950 queries were executed in the warm-up phase, and each query was run 50 times. After that, we run the same permutations 3 more times and measured the execution time of each query. The query response time was computed as the mean response time of executing each query 30 times.
Test Collections Experiments are run on 8 test collections. Each collection includes the Parties and Members data sets and a scaled Proceedings data set ranging from 1 to 100% . Table 2 gives an overview of the sizes of each test collection. Scaling Factor 1% 2% 4% 8% 16% 32% 64% 100% # of triples 494,875 1,027,395 1.906,880 3,851,642 7,554,304 15,129,621 23,341,602 36,542,431 We report on three experiments, relating database size to execution time: (1) time needed to load the test collections ( g. 2), (2) total time needed to execute all the queries in micro benchmarks5 ( g. 3), and (3) query execution time of all the queries on the largest collection ( g. 4).
The y-axes on g. 2 and g. 3 are presented in a log scale, and the numbers represent the loading and query response time in seconds. Appendix E contains larger versions of these plots.
To make the results reproducible, we publish the benchmark data sets, queries and scripts at http://data.politicalmashup.nl/RDF/data/. 5 For each group we summed the execution time of each query in the group.
ParlBench has the proper characteristics of an RDF benchmark: it can be scaled easily and it has a set of intuitive queries which measure di erent aspects of the SPARQL engine.
We believe that ParlBench is a good proxy for a realistic large scale digital publishing application. ParlBench provides real data that encompass major characteristics shared by most of the e-publishing use cases including rich metadata and hierarichal organization of the content into text chunks.
The data set is large enough to perform non-trivial experiments. In addition to the
analytical scenario presented, one can think of several other application scenarios that
can be developed on the same data sets. Due to the many and strong connections of
the benchmark to the Linked Open Data Cloud through the DBpedia links, natural
Linked Data integration scenarios can be developed from ParlBench. The ParlBench
data is also freely available in XML format [
As a future work we will consider the execution of the benchmark on multiple RDF stores and comparison of the results with the ones achieved on Virtuoso.
Another interesting direction for future work could be to extend the set of queries.
Currently, there are only two queries with the OPTIONAL operator, which was proved to
be the reason of the high complexity of the SPARQL language [
The structural elements of the proceedings are connected to Proceedings through the dcterms:hasPart property. The speaker of the speech and the a liated party of the speaker are attached to Speech via the refMember and refParty properties correspondingly.
Vocabularies Relevant existing vocabularies and ontologies to model documents of the parliamentary proceedings fall into two categories. In the rst category there are vocabularies that are too generic, such as the SALT Document Ontology6 or the DoCo, the Document Components Ontology7. They do not provide means to represent such speci c concepts as stage direction or scene. Vocabularies in the second category are too speci c, like the Semantic Web Conference Ontology8 or the Semantic Web Portal Ontology9 which model proceedings of conferences.
We de ned our own RDF vocabulary to model parliamentary proceedings, the Parliamentary Proceedings vocabulary10, and integrated it with other existing vocabularies. To represent biographical information of politicians, we used: BIO: A vocabulary for biographical information11 together with the Friend of a Friend Vocabulary (FOAF) 12 and the DBpedia Ontology13. The Modular Uni ed Tagging Ontology14 (MUTO) was used to represent information about tagged entities of paragraphs. The Dublin Core Metadata Terms15 was used to encode metadata information. 6 http://salt.semanticauthoring.org/ontologies/sdo# 7 http://purl.org/spar/doco/Paragraph 8 http://data.semanticweb.org/ns/swc/swc_2009-05-09.html# 9 http://sw-portal.deri.org/ontologies/swportal# 10 http://purl.org/vocab/parlipro# 11 http://vocab.org/bio 12 http://xmlns.com/foaf/0.1/ 13 http://dbpedia.org/ontology/ 14 http://muto.socialtagging.org/ 15 http://purl.org/dc/terms/ and http://purl.org/dc/elements/1.1/
rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> parlipro: <http://purl.org/vocab/parlipro#> dcterms: <http://purl.org/dc/terms/> dc: <http://purl.org/dc/elements/1.1/> bio: <http://purl.org/vocab/bio/0.1/> foaf: <http://xmlns.com/foaf/0.1/> dbpedia: <http://dbpedia.org/resource/> owl: <http://www.w3.org/2002/07/owl#>
SELECT AVG(?numOfMembers) as ?avgNumOfMembersPerTopic WHERE {{ SELECT COUNT(?member) AS ?numOfMembers WHERE { ?topic rdf:type parlipro:Topic . ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member .
{?topic dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?sd . ?sd rdf:type parlipro:StageDirection .
?sd dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?scene . ?scene rdf:type parlipro:Scene .
?scene dcterms:hasPart ?speech .}}
GROUP BY ?topic}}
SELECT AVG(?numOfSpeeches) as ?avgNumOfSpeechesPerTopic WHERE {{ SELECT COUNT(?speech) AS ?numOfSpeeches WHERE { ?topic rdf:type parlipro:Topic . ?speech rdf:type parlipro:Speech .
{?topic dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?sd . ?sd rdf:type parlipro:StageDirection .
?sd dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?scene . ?scene rdf:type parlipro:Scene .
?scene dcterms:hasPart ?speech .}}
GROUP BY ?topic}}
SELECT AVG(?numOfSpeeches) as ?avgNumOfSpeechesPerDay WHERE {{ SELECT ?date COUNT(?speech) AS ?numOfSpeeches WHERE { ?proc dcterms:hasPart ?topic . ?proc rdf:type parlipro:ParliamentaryProceedings . ?proc dc:date ?date . ?speech rdf:type parlipro:Speech . ?topic rdf:type parlipro:Topic .
{?topic dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?sd . ?sd rdf:type parlipro:StageDirection .
?sd dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?scene . ?scene rdf:type parlipro:Scene .
?scene dcterms:hasPart ?speech .}}
GROUP BY ?date}}
SELECT COUNT(?speech) WHERE { ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member . ?member bio:biography _:bio . _:bio rdf:type bio:Biography . _:bio foaf:gender dbpedia:Female .}
SELECT COUNT(?speech) WHERE { ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member . ?member bio:biography _:bio . _:bio rdf:type bio:Biography . _:bio foaf:gender dbpedia:Male .}
SELECT COUNT(?speech) WHERE { ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member . ?member bio:biography _:bio . _:bio rdf:type bio:Biography . _:bio foaf:birthday ?birthday .
FILTER (year(?birthday) > 1960)}
SELECT COUNT(?speech) WHERE { ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member . ?member bio:biography _:bio . _:bio rdf:type bio:Biography . _:bio foaf:gender dbpedia:Male .
_:bio foaf:birthday ?birthday .
FILTER (year(?birthday) > 1960)} SELECT ?topic ?member COUNT(?speech) as ?numOfSpeeches WHERE {{ SELECT ?topic ?member COUNT(?member) AS ?numOfFemales ?speech WHERE { ?topic rdf:type parlipro:Topic . ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member . ?member bio:biography _:bio . _:bio rdf:type bio:Biography . _:bio foaf:gender dbpedia:Female .
{?topic dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?sd . ?sd rdf:type parlipro:StageDirection .
?sd dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?scene . ?scene rdf:type parlipro:Scene .
?scene dcterms:hasPart ?speech .}} GROUP BY ?topic ?member ?speech} FILTER (?numOfFemales = 1 )}
GROUP BY ?topic ?member F0: What members were born after 1950, their parties and dates of death of exist? SELECT DISTINCT ?member ?party ?birthday WHERE { ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member. ?speech parlipro:refParty ?party . ?member rdf:type parlipro:ParliamentMember . ?member bio:biography _:bio . _:bio rdf:type bio:Biography .
_:bio foaf:birthday ?birthday .
FILTER (year(?birthday) > 1960)} FILTER (year(?birthday) > 1950)
OPTIONAL{_:bio dbpedia-ont:deathDate ?deathDate .}} F1: What gender of politicians who spoke most within a certain timeframe? SELECT ?gender COUNT(?member) AS ?numOfMembers WHERE { ?proc rdf:type parlipro:ParliamentaryProceedings . ?proc dc:date ?date . ?proc dcterms:hasPart ?topic . ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member . ?member rdf:type parlipro:ParliamentMember . ?member bio:biography _:bio . _:bio rdf:type bio:Biography . _:bio foaf:gender ?gender .
{?topic dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?sd . ?sd rdf:type parlipro:StageDirection .
?sd dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?scene . ?scene rdf:type parlipro:Scene .
?scene dcterms:hasPart ?speech .} FILTER (year(?date) > 1995 AND year(?date) < 2005)} GROUP BY ?gender ORDER BY DESC(?numOfMembers)
LIMIT 1
SELECT (?numOfMaleMembers*100)/?numOfMembers WHERE{{ SELECT COUNT(DISTINCT ?memberMale) as ?numOfMaleMembers
COUNT(DISTINCT ?member) as ?numOfMembers WHERE {{ ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member .
?member rdf:type parlipro:ParliamentMember .} UNION{ ?memberMale bio:biography _:bio . _:bio rdf:type bio:Biography .
_:bio foaf:gender dbpedia:Male .
FILTER (sameTerm(?member,?memberMale))}}}}
SELECT (?numOfFemaleMembers*100)/?numOfMembers WHERE{{ SELECT COUNT(DISTINCT ?memberFemale) as ?numOfFemaleMembers
COUNT(DISTINCT ?member) as ?numOfMembers WHERE {{ ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member .
?member rdf:type parlipro:ParliamentMember .} UNION{ ?memberFemale bio:biography _:bio . _:bio rdf:type bio:Biography .
_:bio foaf:gender dbpedia:Female .
FILTER (sameTerm(?member,?memberFemale))}}}} SELECT ?member ?numOfPages WHERE {{ SELECT DISTINCT ?member COUNT(?dbpediaMember) AS ?numOfPages WHERE { ?member rdf:type parlipro:ParliamentMember .
?member owl:sameAs ?dbpediaMember .} GROUP BY ?member}} ORDER BY DESC(?numOfPages)
LIMIT 1 F4: What politician has most number of Wikipedia pages in di erent languages? F5: What speeches are made by politicians without Wikipedia pages? SELECT DISTINCT ?speech ?member WHERE { ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member .
?member rdf:type parlipro:ParliamentMember .
OPTIONAL {?member owl:sameAs ?dbpediaMember .}
FILTER (!bound(?dbpediaMember))}
SELECT ?member COUNT(?speech) as ?numOfSpeeches WHERE { ?member rdf:type parlipro:ParliamentMember . ?speech parlipro:refMember ?member .} GROUP BY ?member ORDER BY DESC(?numOfSpeeches)
LIMIT 10
SELECT ?topic COUNT(?member) as ?numOfMembersSpokeInTopic WHERE {
?topic rdf:type parlipro:Topic . ?topic dcterms:hasPart ?speech . ?speech rdf:type parlipro:Speech . ?speech parlipro:refMember ?member .}
GROUP BY ?topic ORDER BY DESC(?numOfMembersSpokeInTopic)
LIMIT 10
SELECT ?topic COUNT(?speech) as ?numOfSpeeches WHERE { ?topic rdf:type parlipro:Topic . ?speech rdf:type parlipro:Speech . {?topic dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?sd . ?sd rdf:type parlipro:StageDirection . ?sd dcterms:hasPart ?speech .} UNION{ ?topic dcterms:hasPart ?scene . ?scene rdf:type parlipro:Scene . ?scene dcterms:hasPart ?speech .}} GROUP BY ?topic ORDER BY DESC(?numOfSpeeches)
LIMIT 10
SELECT ?date COUNT(?topic) as ?numOfTopics WHERE { ?proc rdf:type parlipro:ParliamentaryProceedings . ?proc dcterms:hasPart ?topic .
?proc dc:date ?date .} GROUP BY ?date ORDER BY DESC(?numOfTopics)
LIMIT 10
For the benchmark evaluation we used a personal laptop Apple MacBook Pro. The operating system running is Mac OS X Lion 10.7.5 x64. The speci cation of the machine is the following: Hardware { CPUs: 2.8 GHz Intel Core i7 (2x2 cores) { Memory: 8 GB 1333 MHz DDR3 { Hard Disk: 750GB Software { OpenLink Virtuoso: Open Source Edition v.06.01.3127 compiled from source for
OS X { MySQL Community Server (GPL) v. 5.5.15 { Scripts (bash 3.2, Python 2.7.3) to scale and upload RDF datasets, to create permutations of queries and run them on Virtuoso. The scripts are available for downloading at http://data.politicalmashup.nl/RDF/scripts/.
Virtuoso Con guration We con gured the Virtuoso Server to handle load of large data sets16.
NumberOfBu ers = 680000 MaxDirtyBu ers = 500000 16 http://www.openlinksw.com/dataspace/doc/dav/wiki/Main/
VirtRDFPerformanceTuning
The RDF Index Scheme remained as it was supplied with the default Virtuoso installation. Namely, the scheme consists of the following indices: { PSOG - primary key. { POGS - bitmap index for lookups on object value. { SP - partial index for cases where only S is speci ed. { OP - partial index for cases where only O is speci ed.
{ GS - partial index for cases where only G is speci ed.
E
Size of proceedings, %