In this paper, we introduce a benchmark to test efficiency of RDF data model for data storage and querying in relation to a concrete dataset. We created Czech DBpedia - a freely available dataset composed of data extracted from Czech Wikipedia. But during creation and querying of this dataset, we faced problems caused by a lack of performance of used RDF storage. We designed metrics to measure efficiency of data storage approaches. Our metric quantifies the impact of data decomposition in RDF triples. Results of our benchmark applied to the dataset of Czech DBpedia are presented.
1.1
Though datasets such as DBpedia can be very useful as pointed out in section 4,
the adoption of these datasets is often limited to semantic web community. One
of the main drawbacks is lack of performance. Data are usually stored as RDF
triples. The performance of main RDF stores (or triple stores) increases as
indicated by several benchmarks [
The main factor that causes RDF stores to be inefficient for certain types of queries is a number of joins the store has to perform in order to evaluate a query. However, in many cases on the web, data is presented in a joined form (e.g. description of one entity and its properties on one page like in a Wikipedia article) and it is split in order to represent it as RDF. We elaborate this process in detail in Section 5. The opened question is, whether this split is necessary. We evaluate this phenomenon and quantify its influence on the dataset performance. Czech DBpedia is used as a representative dataset. 1.2
Contributions – We introduce a comprehensive dataset of machine readable information extracted from Czech Wikipedia. – The presented dataset covers wide variety of areas and preserves the connection of data to corresponding Wikipedia articles. As such can serve as a data integration hub and a source of common identifiers facilitating data integration of diverse data sets. – We show possible applications of this dataset. We focus particularly on specific Czech use cases, where Czech DBpedia is worth than the global one. – We point out the problem of ineffective data representation in case of RDF and quantify its influence on a dataset.
– We provide a SPARQL benchmark for testing data stored in triple stores. 1.3
The rest of the paper is organized as follows. Related work is recalled in Section 2. Section 3 introduces the process of our dataset creation and Section 4 describes possible use cases of the dataset. Section 5 describes methods used to store RDF data and introduces our metrics measuring efficiency on a concrete dataset. Experimental results are presented in Section 6. The paper is summarized in Section 7. 2
The whole philosophy of DBpedia and technical details are provided in [
Additionally, based on our dataset, we introduce metrics to measure the
performance of data storage. There are several benchmarks measuring the
performance of RDF data stores using SPARQL queries, similarly to our work.
The suite of benchmarks for comparing the performance of SPARQL endpoints
- Berlin SPARQL Benchmark [
All the mentioned benchmarks use same metrics to measure the performance. The basic metrics are Queries per Second (QpS), Query Mixes per Hour (QMpH) and Overall Runtime (oaRT). But all these metrics are heavily dependent on a hardware configuration and overall system conditions of a test run. In Section 5.2, we introduce metrics that are more qualitative and quantitative characteristics of the tested dataset according to the used storage approach. Thus the results of our benchmark are completely independent of a concrete hardware configuration.
In order to develop our metrics, we considered various approaches to RDF
data storage presented by major triple stores Jena [
In order to obtain raw data from Wikipedia, we use the DBpedia Extraction
Framework [
Various information can be obtained from Wikipedia pages. It is quite easy to get labels of entities and extract their connections by analysis of links between corresponding Wikipedia articles. However, the core of the extraction process is the retrieval of information contained in so called infoboxes. An example of such an infobox in a Wikipedia article is shown in Figure 1.
In case of the Czech DBpedia, we use following extractors provided by the extraction framework: Label Extractor (extracts labels of entities from titles of corresponding articles), Geo Extractor (extracts geographic coordinates), Page
Links Extractor (extracts internal links between DBpedia instances from the internal pagelinks between Wikipedia articles), Wiki Page Extractor (extracts links to corresponding articles on Wikipedia), Infobox Extractor (extracts all properties from all infoboxes) and Mapping Extractor (extracts structured data based on hand-generated mappings of Wikipedia infoboxes to the DBpedia ontology).
It is important to note the difference between Infobox Extractor and Mapping Extractor. Consider the source code from Figure 1. The Infobox Extractor extracts this information as it is written in the source code. Property names are not cleaned, there is no consistent ontology for the infobox dataset. Thus generating RDF triples like: <http://cs.dbpedia.org/resource/Albert_Einstein> <http://cs.dbpedia.org/property/misto_narozeni> <http://cs.dbpedia.org/resource/Ulm> .
Unfortunately, infoboxes on Wikipedia are inconsistent and it is usual that same property is described in many different ways. Someone can call the same property misto narozeni (placeOfBirth), whereas someone else might use puvod (origin), or narozeni misto (birthPlace).
The answer to these difficulties is the Mapping Extractor which uses hand written rules that map different patterns used in Wikipedia infoboxes to a consistent ontology of DBpedia.
For our dataset we generated rules covering 60 most important entities (e.g.
cities, politicians, actors, writers). Totally, there are currently about 109 Infobox
templates on Czech Wikipedia that can be potentially mapped to DBpedia
ontology. This means so far we have mapped more than half of them. Mappings
can be edited via the Mappings Wiki [
In this section, we want to point out the advantages of a local clone of DBpedia compared to the English one.
Similarly to local Wikipedias, local DBpedias usually provide more comprehensive information about specific local entities, like geographical data (smaller Czech cities, mountains, rivers, lakes), data about important persons (Czech politicians, movie directors, writers, etc.).
As such, this dataset may serve as a base for various mashup application or automatic data processing tools. Apart from the range of locally specific data, the language of entries and their direct connection to Czech Wikipedia pages might be an advantage too.
Thanks to tens of manually created mapping rules, Czech DBpedia is a good ontology mapping dictionary as well. It may serve as a central hub providing a set of common identifiers together with basic properties of identified entities.
All machine readable data on DBpedia have a direct connection to corresponding Wikipedia articles, where the information is presented in an unstructured way, usually as a plain text. Thus Czech DBpedia might serve as a testing dataset for various natural language processing and information retrieval approaches focusing on Czech language. 5 5.1
The RDF data model represents data as statements about resources using a graph connecting resource nodes. An RDF statement has the form of a triple consisting of subject, predicate and object. In the following text, unless otherwise stated, under subject, predicate and object, we understand the appropriate part of an RDF triple.
The majority of RDF data storage solutions including Jena [
This query results in following SQL query over triples table, invoking three joins:
FROM triples_table AS T1, triples_table AS T2, triples_table AS T3 WHERE T1.subject = T2.subject AND T1.property = ’director’ AND T1.object = ’"Jan Sveˇr´ak"’ AND T2.subject = T3.subject AND T2.property = ’filmed’ AND T2.object = ’"1996"’ AND T3.property = ’name’
Instead of storing whole identifiers and literals directly in the triples table, Oracle, Virtuoso and Sesame replace strings with integer identifiers, so that the data is normalised in two tables. Whereas Jena takes the trade off - space for speed - and keeps strings directly in the triples table.
In order to minimize the count of inefficient join operations that are invoked by each statement used in SPARQL query, various optimizations were proposed.
Property tables first introduced by authors of Jena [
A variant of property tables are property-class tables, where clusters are created based on RDF types of subjects.
A similar approach to the property table data structure was introduced in [
A different approach is vertical partitioning described in [
The aim to minimize the impact of joins on the query execution leads us to define metrics that quantify this impact on a particular dataset.
In the following text, we use the term shallow properties to label properties such that there is at least one resource appearing with them in an object position that does not appear in a subject position in another statement.
Contrary, by deep properties we mean properties such that there is at least one resource appearing with them in an object position that appears in a subject position in at least one another statement.
Joinability measures an average count of shallow properties on a subject. If shallow properties were stored in one table (as columns of the table), there would not be a need to join the data at all. Shallow properties result in an avoidable subject-subject join in that case. Joinability is defined as follows: j = cs |S|
Where cs is the count of statements, where a property plays a role of a shallow property and |S| is the count of distinct subjects in the dataset.
Linkability is an average count of deep properties on a subject. The occurrence of deep properties may result in an object-subject join, if we query for properties of an object as well. Linkability is defined in the following way:
Another interesting characteristic of a dataset is an average indegree of an object and outdegree of a subject. Especially outdegree of a subject indicates, how many joins could be potentially saved, if all the properties of a particular subject were stored in one row. Note that sum of an average joinability and linkability gives an average outdegree of subjects in a dataset. 6 6.1
For the evaluation, we used the dataset of Czech DBpedia. It is smaller than the dataset of English DBpedia, so it is feasible to run some more demanding queries that are difficult to run in reasonable time on the English one. However, still Czech DBpedia represents a comprehensive dataset, with similar characteristics to the English one. In this section, we provide some basic characteristics of data that can be extracted from Czech Wikipedia and compare it to the English Wikipedia.
l = cd |S| w = j l
Where cd is the count of statements, where a property plays a role of a deep property and |S| is the count of distinct subjects in the dataset.
Czech DBpedia contains totally 12 402 513 RDF statements. About 251 877
RDF statements are extracted based on our hand written mappings, whereas
the English DBpedia composes of about 17,5 million RDF statements based on
mappings (according to the last dataset release report [
Data is extracted from 220 000 articles on Czech Wikipedia. English Wikipedia has about 3 800 000 articles which is more than 17 times bigger dataset.
There are 704 760 distinct subjects that have some property in the current Czech dataset.
Additionally, we counted entities. Under entities, we mean real world concepts. Usually, each entity corresponds to a Wikipedia article that describes it. Thus total count of entities should correspond to 220 000 Wikipedia articles. But in reality the count of identified entities is heavily dependent on a coverage of hand written mapping rules. Entities have commonly a type. Counts of most common entities compared to the English dataset are provided in Table 1. It is remarkable that in case of cities, the count of entities in Czech dataset is closer to the English dataset than in other cases (it is more than 34% of the count of cities in English dataset). This points to the fact that information about cities are well elaborated on Czech Wikipedia and also good mappings are provided to transform it to DBpedia. The English DBpedia presents actually much more countries than their real count in the world. This fact is caused by a vague definition of a country on Wikipedia. For example a self-governing British Overseas Territory Falkland Islands is considered to be a country as well. In this section, we provide some characteristics, we measured on the dataset of Czech DBpedia with respect to metrics introduced in Section 5.2. Due to the lack of space in this paper, we present the actual benchmark on a separate web page5.
In Figure 2, we compare count of distinct subjects with at least one shallow property and at least one deep property. We can see that almost all subjects 5 The whole benchmark used to obtain presented results composes of SPARQL queries and basic scripts to parse results of these queries. It is available for download at http://research.i-lasek.cz/semantic-web/joinability-benchmark/ have a shallow property as well as a deep property. There were less than 6 000 subjects having all properties shallow. These were in most cases operational and meta data information rather than real entities such as cities, persons etc.
In Figure 3, we further investigated overall count of distinct shallow properties and distinct deep properties. It is remarkable that both sets have a common intersection. According to the results, almost all properties are in some cases shallow. While some of them play a role of a deep property as well. This is legal, because a property might connect a subject with an object, that is not further described in the dataset, while in another case another connected object is a subject in another statement. Also sometimes, data is messy and same property connects subject with a literal value and at the same time connects it with an object with further properties. 1 697 out of 6 714 properties played a role of deep properties in at least one statement.
In Figure 4, we compare counts of objects that do not play a role of a subject in any other statement (result in a shallow connecting property), with objects that play a role of subjects in at least one another statement (result in a deep connecting property).
Afterwards, we measured counts of non distinct shallow and deep properties and counted Linkability and Joinability. For Czech DBpedia, these are: j = 8, 92 and l = 8, 67.
Finally, we evaluated average outdegree of subjects (for Czech DBpedia it is approximately 17,6) and average indegree of objects (approximately 5,0). For the quantification of join impact, the outdegree of subjects is especially important. This means that, while querying DBpedia for all properties of an entity, a query results in average in almost 18 self-joins, if we consider the triples table approach described in Section 5.1. The distribution of subjects outdegree is displayed in Figure 5.
We introduced Czech DBpedia and briefly described the process of creation of this dataset. The main idea was introduced: On the web, huge amount of data is presented in a joined from (e.g. infobox tables on Wikipedia). However, often this data is split in order to present it as RDF. Afterwards it has to be joined again, while querying RDF data storage, which is a demanding operation. We designed metrics to measure efficiency of data storage approaches, according to the join demands. We introduced a benchmark to test efficiency of RDF data model for data storage and querying in relation to a concrete dataset. Finally, we presented results of this benchmark applied to a dataset of Czech DBpedia. The benchmark is publicly available on the web, so anyone can use it to test any dataset accessible via a SPARQL endpoint. 7.1
In our future work, we plan to design an efficient RDF repository. Based on our
current findings, we intend to use property tables. Our modification is that we
use a variant of the algorithm for creating concepts (in the sense of Conceptual
lattices (see [