The advancements and popularity of Semantic Web technologies in the last decades have led to an exponential adoption and availability of Web-accessible datasets. While most solutions consider such datasets to be static, they often evolve over time. Hence, eficient archiving solutions are needed to meet the users' and maintainers' needs. While some solutions to these challenges already exist, standardized benchmarks are needed to systematically test the diferent capabilities of existing solutions and identify their limitations. Unfortunately, the development of new benchmarks has not kept pace with the evolution of RDF archiving systems. In this paper, we therefore identify the current state of the art in RDF archiving benchmarks and discuss to what degree such benchmarks reflect the current needs of real-world use cases and their requirements. Through this empirical assessment, we highlight the need for the development of more advanced and comprehensive benchmarks that align with the evolving landscape of RDF archiving.
the field, current benchmarks do not suficiently capture emerging challenges faced by archiving systems. We use this finding to derive a set of requirements, that we believe, are essential for benchmarks to advance research and development of RDF archives.
The remainder of this paper is organized as follows. First, we discuss the current state of RDF archiving research and relevant benchmarks in Section 2. Second, in Section 3, we discuss the shortcomings of current RDF archiving benchmarks and our recommendations and requirements for the future. Finally, Section 4 concludes the paper.
We now provide a brief survey of the available RDF archiving systems as well as of existing languages and SPARQL extensions designed for RDF archives. Furthermore, this section delves into existing benchmarks tailored for assessing the performance of RDF archiving systems.
RDF archiving, at its core, consists of storing and querying the entire evolution history of an
RDF graph. This has proven to be a challenging task due to the additional temporal dimension
compared to traditional RDF stores. While the design of eficient indexing and querying systems
for RDF archives is still an ongoing efort, multiple approaches have been proposed throughout
the years [
In contrast to conventional RDF, the existence of multiple versions within an RDF archive
introduces the need for novel query types that can be hardly expressed in standard SPARQL.
Some approaches propose the extension of SPARQL to support temporal queries, i.e., by
specifying a timestamp or interval in which the query results should hold [
Benchmarks play a crucial role in guiding the development of systems by facilitating their
evaluation and enabling comparisons with existing systems in terms of implementation and
design. Due to being a relatively new area in RDF data management, we only account for three
benchmarks tailored for RDF archiving in the literature: EvoGen [
EvoGen [
BEAR [
SPBv [
In this section, we examine the qualities and features that a benchmark for RDF archiving should strive to possess. We propose that benchmarks for RDF Archives should strive for three main overarching qualities, namely reproducibility, realism, and configurability . Reproducibility represents the ease at which the benchmark results can be shared and reproduced by others. Realism is about how the benchmark setting, both in the choice of dataset and query loads, models or emulates the real world. Configurability represents the ability of the benchmark to propose workloads of various sizes, relevant for a wide range of system configurations and use cases. We further detail our recommendations of a concrete implementation of the aforementioned qualities by first detailing the choice of data. We then will discuss the design of query workloads, and finally, we discuss whether existing benchmarks fulfill those requirements.
The choice of data is an important aspect when designing a benchmark. Current benchmarks
use either a configurable generator for synthetic data [
Most importantly, a benchmark should cover diferent, realistic, scaling options. In the RDF archiving world, the scaling options do not only cover diferent data sizes, but also the history’s size, i.e. the number of versions and the magnitude of changes within each version. Generator-based benchmarks should provide users with all the necessary scaling parameters, while real-world-based benchmarks should ofer diferent datasets scaling along those axes.
Early RDF archiving systems could be adequately tested with single triple pattern queries, but
contemporary archiving benchmarks should prioritize comprehensive SPARQL query
workloads. We believe that eficient support for full SPARQL represents a major challenge that RDF
archiving systems currently need to solve. Consequently, in order to fulfill our realism
requirement, benchmarks should provide comprehensive assessment of those capabilities, guiding the
development of existing and new systems. Benchmark query workloads should be carefully
designed to align with real-world use cases. Following recommendations from the LDBC [
Finally, the lack of an accepted standard to formulate archiving queries into SPARQL is a
major brake for the design of benchmark queries. Addressing this issue necessitates a dedicated
standardization efort, drawing inspiration from the RDF stream community, and the RSP-QL
standardization [
Table 1 summarizes the characteristics of the existing RDF archiving benchmarks. Among the
available benchmarks, two of them rely on synthetic data generated through a data generator.
Generator-based systems fulfill the configurability criteria easily due to their nature, but may
fall short of also proving their realism, while their reproducibility is dependent on the sharing of
the exact parameters and random seed. Both EvoGen [
In this paper, we presented the current state in RDF archiving systems and benchmarks. We have proposed a set of requirements that benchmarks should have in order to contribute to the advancement of the field. We showed that among the only three available benchmarks for RDF archiving systems, none of them proposes a satisfactory set of features. This ranges from a general lack of realism w.r.t. the real world, lack of SPARQL support, or concerns with reproducibility. We see several areas open for future work. First, precisely defining the semantics and syntax of SPARQL archive queries would benefit greatly to the wider RDF community. This would open the door for standardized support across various RDF stores and research systems. Secondly, benchmarks relevant to the modern challenges faced by RDF archiving applications and systems are needed to guide and evaluate eforts in that area. We believe that this is paramount to current development eforts of fully-fledged RDF archiving systems. This research is partially funded by the Poul Due Jensen Foundation and the Independent Research Fund Denmark (DFF) under grant agreement no. DFF-8048-00051B, the TAILOR Network (EU Horizon 2020 research and innovation program under GA 952215), and the Research Foundation – Flanders (FWO) (1274521N).