In ontology-based data access (OBDA), the aim is to provide a highlevel conceptual view over potentially very large (relational) data sources by means of a mediating ontology. The ontology is connected to the data sources through a declarative specification given in terms of mappings that relate each (class and property) symbol in the ontology to an (SQL) view over the data. Although prototype OBDA systems providing the ability to answer SPARQL queries over the ontology are available, a significant challenge remains: performance. To properly evaluate OBDA systems, benchmarks tailored towards the requirements in this setting are needed. OWL benchmarks, which have been developed to test the performance of generic SPARQL query engines, however, fail at 1) exhibiting a complex real-world ontology, 2) providing challenging real world queries, 3) providing large amounts of real-world data, and the possibility to test a system over data of increasing size, and 4) capturing important OBDA-specific measures related to the rewriting-based query answering approach in OBDA. In this work, we propose a novel benchmark for OBDA systems based on a real world use-case adopted in the EU project Optique. We validate our benchmark on the system Ontop, showing that it is more adequate than previous benchmarks not tailored for OBDA.
In ontology-based data access (OBDA), the aim is to provide a high-level conceptual
view over potentially very large (usually relational) data sources by means of a mediating
ontology. Queries are posed over such conceptual layer and then translated into queries
over the data layer. The ontology is connected to the data sources through a declarative
specification given in terms of mappings that relate each (class and property) symbol in
the ontology to an (SQL) view over the data. The W3C standard R2RML [
To properly evaluate the performance of OBDA systems, benchmarks tailored towards the requirements in this setting are needed. OWL benchmarks, which have been developed to test the performance of generic SPARQL query engines, however, fail at 1) exhibiting a complex real-world ontology, 2) providing challenging real world queries, 3) providing large amounts of real-world data, and the possibility to test a system over data of increasing size, and 4) capturing important OBDA-specific measures related to the rewriting-based query answering approach in OBDA. For instance, in the Berlin SPARQL ? This paper is supported by the EU under the large-scale integrating project (IP) Optique (Scalable
End-user Access to Big Data), grant agreement n. FP7-318338.
Benchmark [
In this work, we propose a novel benchmark for OBDA systems based on the
Norwegian Petroleum Directorate (NPD), which is a real world use-case adopted in the
EU project Optique [
We present the NPD benchmark1 in Section 2, discuss our data generator in Section 3, validate our benchmark in Section 4, and conclude in Section 5. 2
The Norwegian Petroleum Directorate [
The Ontology. The ontology contains OWL axioms specifying comprehensive information about the underlying concepts in the dataset. Since we are interested in benchmarking OBDA systems that are able to rewrite SPARQL queries over the ontology into FOL queries (therefore, SQL queries) that can be evaluated by a relational DBMS, we concentrate here on the OWL 2 QL profile [20] of OWL, which guarantees FO-rewritability of
unions of conjunctive queries (see, e.g., [
The Mappings. The mapping consists of 1190 assertions mapping a total of 464 among classes, objects properties, and data properties. The SQL queries in the mappings, in terms of their direct DATALOG translation, count an average of 2:6 rules, with 1:7 joins per rule. We observe that the mappings have not been optimised to take full advantage of an OBDA framework, e.g., by trying to minimize the number of mappings that refer to the same ontology class or property, so as to reduce the size of the SQL query generated by unfolding the mapping. 3
The generator produces data according to certain features of an input training database. The algorithm is not specific for NPD, and it can in principle be applied to every database. We put special care in making the generation process fast, so as to be able to generate very large datasets. The algorithm starts from a non-empty database D. Given a desired increment f > 0, it generates a new database such that jT 0j = jT j (1 + f ), for each table T of D that has to be incremented (where jT j denotes the number of tuples of T ). Duplicate Values Generation. Values in each column T:C of a table T 2 D are generated with a duplicate ratio (jjT:Cjj jT:Cj)=jjT:Cjj, where jjT:Cjj (resp., jT:Cj) denotes the cardinality of the column T:C under the multi-set (resp., set) semantics. A duplicate ratio “close to 1” indicates that the content of the column is essentially independent from the size of the database, and it should not be increased by the data generator. Fresh Values Generation. For each non-string column T:C over a totally ordered domain, the generator chooses values from the interval I := [min(T:C); max(T:C)]. If the number of values to insert exceeds the number of different fresh values that can be chosen from the interval I, then values greater than max(T:C) are allowed.
Chase Cycles. The data generation is done respecting foreign keys. Let T1 ! T2 denote the presence of a foreign key from table T1 to table T2. In case of a cycle T1 ! T2 ! ! Tk ! T1, inserting a tuple in T1 could potentially trigger an infinite number of insertions. By analyzing the input database, the generator prevents an infinite chain of insertions while ensuring that no foreign key constraint is violated.
Geometric Types. The NPD database makes use of geometric datatypes available in MYSQL. Some of them come with constraints, e.g., a polygon is a closed non-intersecting line composed of a finite number of straight lines. For each geometric column in the database, the generator first identifies the minimal rectangular region of space enclosing all the values in the column, and then it generates values in that region. Generator Validation. We aim at producing synthetic data that approximates the realworld data well. By considering the mapping assertions, it is possible to establish the expected growth of the ontology elements w.r.t. the growth of the database. For our analysis we focus here on those ontology elements for which the expected growth has to be either linear (w.r.t. the growth of the database) or absent. We tested this way 138 concept names, 28 object properties and 226 data properties.
Table 3 reports the results of the aforementioned analysis. The first column indicates the type of ontology elements being analyzed, and the specified increment f (e.g., “class npd2” refers to the population of classes for the database incremented with factor f = 2). The columns “avg err” show the averages of the errors between the expected growth and the observed growth, in terms of percentage of the expected growth. The remaining columns report the number and percentage of ontology elements for which the error was greater than 50%.
Concerning concepts, our generator seems to behave well. This is because concepts
are usually populated starting from primary keys in the database, or by simple queries
that do not involve the Cartesian product of two or more tables (in this last case the
growth would be quadratic). More attention, instead, has to be paid when populating
object or data properties. Our data generator, indeed, considers the distribution of the
elements in a column-wise manner, and ignores the distribution of the actual relations in
the ontology. Observe that, in order to solve this problem, analyzing the distribution at the
level of tuples in relations might in general be insufficient, since object/data properties
can be populated with columns from different tables. On the other hand, a more accurate
approximation working at the level of the triples in the ontology [24] might be unfeasible,
because of the well-known view update problem [
Finally, the performed tests and the data reported in Table 3 indicate that our heuristic generator substantially outperforms a pure random generator. 4
We ran the benchmark on the Ontop system2 [21], which, to the best of our knowledge, is the only one fully implemented OBDA system that is freely available. MYSQL was used as underlying relational database system. The hardware consisted of an HP Proliant server with 24 Intel Xeon X5690 CPUs (144 cores @3.47GHz), 106GB of RAM and a 1TB 15K RPM HD. The OS is Ubuntu 12.04 LTS. Due to space constraints, we present the results for only one running client.
Results were obtained with a configuration that does not consider anonymous individuals. This is motivated by the fact that none of the queries for which this kind of reasoning would give additional answers (i.e., queries with tree witnesses) could be rewritten, and executed, within the given timeout (1 min.).
In order to test the scalability of the system w.r.t. the growth of the database, we used the data generator described in Section 3 and produced several databases, the largest being approximately 1500 times bigger than the original one (“NPD1500” in Table 4, 117 GB of size on disk).
Table 4 refers to a mix of the 7 easiest queries from the initial query set, for which the unfolding of the mapping produces exactly one non-union SQL query. The mix is executed 10 times, each time with different parameters at the filter conditions, so that the effect of caching is minimized, and statistics were collected for each execution. These are the only queries which display a response time—that is, the sum of the query execution time (avg(ex time)) and the time spent by the system to display the results to the user (avg(out time))—that is less than one minute for every database in the benchmark. Results are in milliseconds, and columns qmpH and avg(res size) refer to query mixes per hour and the average number of results for queries in the mix, respectively.
Table 5 contains results for the 5 hardest queries. Each query was run once with a timeout of 1 minute on the response time. Observe that the response time tends to grow faster than the growth of the underlying database. This follows from the complexity of the queries produced by the unfolding step (column #unfolding indicates the number of union operators in the produced sql query), which usually contain several joins (remember that the worst case cardinality of a result set produced by a join is quadratic in the size of the original tables). Column NPD10 RAND witnesses how using a purely random data generator gives rise to datasets for which the queries are much simpler to evaluate. This is mainly due to the fact that a random generation of values tends to decrease the ratio of duplicates inside columns, resulting in smaller join results over the tables [23]. Hence, purely randomly generated datasets are not appropriate for benchmarking. 5
The benchmark proposed in this work is the first one that thoroughly analyzes a
complete OBDA system in all significant components. So far, little or no work has been
done in this direction, as pointed out in [19], since the research community has mostly
focused on rewriting engines. Thanks to our work, we have gained a better understanding
of the current state of the art for OBDA systems: first, we confirm that the unfolding
phase is the real bottleneck of modern OBDA systems [
We conclude by observing that for a better analysis it is crucial to refine the generator in such a way that domain-specific information is taken into account, and a better approximation of real-world data is produced. 19. Mora, J., Corcho, O.: Towards a systematic benchmarking of ontology-based query rewriting systems. In: Proc. of the 12th Int. Semantic Web Conf. (ISWC 2013). Lecture Notes in Computer Science, vol. 8218, pp. 369–384. Springer (2013) 20. Motik, B., Fokoue, A., Horrocks, I., Wu, Z., Lutz, C., Cuenca Grau, B.: OWL 2 web ontology language profiles. W3C Recommendation, World Wide Web Consortium (Oct 2009), available at http://www.w3.org/TR/owl-profiles/ 21. Rodriguez-Muro, M., Kontchakov, R., Zakharyaschev, M.: Ontology-based data access: Ontop of databases. In: Proc. of the 12th Int. Semantic Web Conf. (ISWC 2013). Lecture Notes in Computer Science, vol. 8218, pp. 558–573. Springer (2013) 22. Skjaeveland, M.G., Lian, E.H.: Benefits of publishing the Norwegian Petroleum Directorate’s FactPages as Linked Open Data. In: Proc. of Norsk informatikkonferanse (NIK 2013). Tapir (2013) 23. Swami, A., Schiefer, K.: On the estimation of join result sizes. In: Proc. of the 4th Int. Conf. on Extending Database Technology (EDBT 1994). Lecture Notes in Computer Science, vol. 779, pp. 287–300. Springer (1994) 24. Wang, S.Y., Guo, Y., Qasem, A., Heflin, J.: Rapid benchmarking for semantic web knowledge base systems. In: Proc. of the 4th Int. Semantic Web Conf. (ISWC 2005). Lecture Notes in Computer Science, vol. 3729, pp. 758–772. Springer (2005)