Performance benchmarks are invaluable for evaluating and comparing federated query processing systems, but it is hard to design benchmarks that are both realistic and informative about the systems being tested. In this paper we present GeoFedBench, a benchmark that has been obtained from an actual, practical application of geospatial and linked data querying and uses GeoSPARQL constructs that challenge all phases of federated query processing. The benchmark is publicly available as part of the Kobe suite.
Performance benchmarks are invaluable for evaluating and comparing systems, but designing benchmarks is subject to considerations that are di cult to satisfy simultaneously. One potential tension is the creation of a realistic benchmark that accurately re ects how the benchmarked systems will behave in real-world use cases against the design of a benchmark that is informative with respect to system characteristics we know in advance that we need to test and measure.
Given the above, we are excited to present a benchmark that has been obtained from an actual, practical application of geospatial and linked data querying. The benchmark federates a database of Earth Observation data about land usage and a database of ground observations about land usage, to search for pairs between them that simultaneously satisfy geospatial and thematic (land usage) constraints (Section 2).
Besides being extracted from a real work ow in the Earth Observation domain, the benchmark queries use GeoSPARQL constructs that challenge all ? Copyright (c) 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4/0). phases of federated query processing, from source selection to query planing and execution. Besides a detailed analysis of the queries, we also present emprical tests demonstrating that the benchmark is both challenging but feasible (Section 3). Finally, we recap and conclude (Section 4). 2
Detailed land usage data is crucial in many applications, ranging from formulating agricultural policy and monitoring its execution, to conducting research on climate change resilience and future food security. Land usage can be inferred from Earth Observation images or collected through self-declaration, but in either case needs to be validated against land surveys. The standard approach for this validation is to match each instance in the land survey dataset (GPS points) with the nearest land parcel (a GIS shape) and compare the crops observed in the survey against the crops declared or inferred for the matching parcel.
Although conceptually straightforward, in operational scenarios this rule can be misleading. Ground observations are geo-referenced to a point on the road adjacent to the eld, which is often ambiguous in agricultural areas with several adjacent parcels; further exacerbated by GPS accuracy. However, a more sophisticated (and also computationally demanding) approach can estimate the error rate of the land usage data: for every survey point there must be at least one parcel with the same label in reasonable proximity; otherwise at least one nearby parcel is mis-labelled (although we cannot automatically infer which one).
For our benchmark, we use the Invekos dataset, the Austrian administration's Land Parcel Identi cation System with owners' self-declaration about the crops grown in each parcel, compared against the observations from the 2018 Land use and cover area frame statistical survey (Lucas). Table 1 gives more details about these datasets.
Besides geospatial processing, using these datasets also introduces a data integration aspect to our benchmark. Speci cally, Lucas annotations follow the Land Cover Classi cation System (LCCS) whereas Invekos follows its own codelist of 212 crop types. There is no one-to-one mapping between instance labels (e.g., Invekos grassland can be Lucas E10, B55, or E30, while Lucas B13 can be Invekos spring barley or winter barley ).
All triples Geospatial triples Thematic triples Lucas 30,379 4,325 26,054 https://esdac.jrc.ec.europa.eu/projects/lucas Invekos 14,036,799 2,005,257 12,031,542 https://www.data.gv.at/katalog/dataset/e21a731f-9e08-4dd3-b9e5-cd460438a5d9
In order to estimate the reliability of the Invekos dataset, we used queries that, for each given Lucas instance, check if: (Q1) the closest Invekos instance is under 10 meters away and their crop labels match; (Q2) the closest Invekos instance is under 10 meters away and their crop labels do not match; or (Q3) there is no Invekos instance within 10 meters.
Since geo-linked data vocabularies link instances with a geometry object (which then has as an attribute the actual shape), these queries (and geoSPARQL queries in general) challenge FILTER optimizers because it presents them with comparisons between variable groundings (as opposed to constant values), and because these comparisons are non-standard extensions (the geospatial extensions of GeoSPARQL).
In most benchmarks, lters are either not present at all [LUBM, 2] or only
have unary functions or comparisons against constants [FedBench, 5] that can
always be pushed into one data source. LargeRDFBench [
Properties of standard vocabularies, which can appear possibly in all sources of a federation, present another challenge in the e cient evaluation of a query. When evaluating a triple pattern that contains a property such as rdf:type or owl:sameAs the source selector is prone to overestimate the set of relevant sources, thus increasing both network tra c and the overall query processing time. Current benchmarks already contain such commonly used properties. But GeoFedBench stresses source selections more on this direction by exploiting a query characteristic that appears frequently in Geospatial data; a resource ?x is linked with its geometry representation ?wkt using chains of known properties of the form ?x geo:hasGeometry ?g . ?g geo:asWKT ?wkt, where all members of the chain usually appear in the same dataset. The federation engine is tested on distinguishing which geospatial triple patterns refer to which dataset, thus avoiding to fetch redundant bindings for the variable in the middle of the chain.
Finally, the complex nature of our queries challenges query planning. Current benchmarks usually contain simple queries consisting only joins between triple patterns and FILTER operations, or some additional operators such as UNION, ORDER, LIMIT, etc. In GeoFedBench, Q1 and Q2 use a subquery for discovering the closest Invekos instance. Also, Q2 and Q3 use negation, in the form of the FILTER NOT EXISTS operator to check that there does not exist and matching Invekos instance. Both subqueries and negation are not present in any of the currently existing federated SPARQL benchmarks.
To demonstrate that the benchmark is feasible but challenging, we tested on
Semagrow [
We presented GeoFedBench, a benchmark for federated geospatial query processing. GeoFedBench is based in openly available datasets and queries challenge all phases of federated query processing. The benchmark is distributed as part of the benchmark suite of the KOBE Open Benchmark Engine, available from https://github.com/semagrow/kobe
This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825258. Please see http://earthanalytics.eu for more details.