Collaborative mechanisms allow benchmarks to be updated continuously and adjust to the changing requirements and new use cases. This paradigm is employed for example in the field of machine learning, but up until now there were no examples of truly open and collaborative benchmarks for RDF systems. In this demo paper we present the collaboration functionalities of RiverBench, an open, multi-task RDF benchmark suite. Owing to its fully open and community-driven design, RiverBench allows any researcher or practitioner to submit a new dataset or benchmark task, report performed benchmark runs, and edit any resource in the suite. RiverBench's collaboration system is itself based on RDF and Linked Data mechanisms, and every resource in the suite has machine-readable RDF metadata. The showcased functionalities together make up a first-of-a-kind fully open and collaborative RDF benchmark suite. These features are meant to encourage other researchers to contribute to RiverBench, and make it a long-term project sustained by the community.
Benchmarking Resource Description Framework (RDF) systems is a popular topic in the Semantic Web and Knowledge Graph research communities. Benchmarks not only allow one to compare how diferent systems perform in certain scenarios, but they also help drive innovation, by identifying gaps in performance and functionalities. Most often, a benchmark task is defined in a research paper, with the dataset(s) and benchmark code attached as supplementary resources. However, it is very hard for such benchmarks to evolve over time, for example, by adding new datasets or metrics contributed by the community. Furthermore, the dispersed nature of these benchmarks makes it dificult to follow consistent standards in benchmark specification, dataset distribution, and result reporting. This in turn reflects negatively upon benchmark reproducibility and slows down progress in the field. We postulate that these issues could be at least partially overcome with an open, collaborative benchmark suite which could naturally evolve over time to suit the changing needs of the community, while maintaining consistent standards.
Although many benchmarks and benchmark suites for RDF systems were proposed, few of them
incorporate any collaborative aspects. The most prominent example here is the Linked Data Benchmark
Council (LDBC), which maintains several graph benchmarks, including RDF-based workloads (e.g., the
Social Network Benchmark [
The idea of collaborative benchmarks is more popular in other areas of computer science, especially
in machine learning (ML) and natural language processing (NLP). For example, the GLUE benchmark
for natural language understanding [
Contribution. In this demo paper we showcase the recently-developed collaborative features of
RiverBench [
RiverBench collects both benchmark tasks and datasets. Although RiverBench focuses on streaming use cases (e.g., streaming RDF graphs over the network), it also includes non-streaming tasks, such as loading RDF data into a triple store, or serializing an RDF graph to bytes. Benchmark profiles group datasets with shared technical characteristics (e.g., datasets with RDF triples only, no quads), to make dataset selection easier. Every resource in RiverBench has an HTML documentation page and machine-readable RDF metadata, available via the content negotiation mechanism. The suite makes extensive use of Continuous Integration / Continuous Deployment (CI/CD) scripts and GitHub’s free infrastructure to automate most tasks.
Benchmark datasets are added to the suite through a public community process, visualized in Figure 1. Any practitioner or researcher is invited to submit their dataset, as long as it meets basic technical requirements and can be published under a permissive license. Firstly, the contributor fills out the dataset proposal form on GitHub, giving details about the dataset’s source, license, original use case, etc. The proposal is then reviewed by RiverBench curators, who check if the dataset meets the publiclyavailable criteria for new datasets (e.g., open license, properly specified authorship, suficient size, clear use case) and if it would be a valuable addition. The curators are members of the RDF benchmarking community, who besides reviewing dataset proposals also have a deciding voice in major decisions shaping RiverBench. As of November 2024, there are four registered curators from three diferent institutions1. After the proposal is accepted by the curators, a technical administrator creates a new GitHub repository for the dataset, and instructs the contributor on how to proceed. The contributor then uploads the dataset for processing and fills out a metadata file in the Turtle language, describing the dataset in a structured manner. The dataset and the metadata are then automatically processed by CI/CD: validating its contents, re-packaging the dataset in several formats, adding more metadata, generating documentation pages, and publishing it on the website under a permanent URL (PURL).
Proposing new benchmark tasks follows a very similar public community process. The contributor also fills out a proposal form on GitHub, which is reviewed by the curators who check if the task
ttl Contributor submits a public dataset proposal
Curators review the proposal and give feedback
Technical admin creates a git repository
Contributor uploads dataset source files and metadata
Validate
Package Add metadata Generate docs
Publish CI/CD processes and publishes the dataset html rdf
[PURL] The dataset is ready for reuse meets the pre-defined criteria. Afterwards, the contributor is asked to write down the description and metadata of the task in a Turtle file. The new task is then automatically processed by CI/CD (validation, packaging, documentation generation, publishing).
The source code of tasks, datasets, profiles (groups of datasets), documentation pages, and other resources of RiverBench are hosted on GitHub as Markdown or Turtle files in one of the several repositories. To make editing these resources as easy as possible, every page on the RiverBench website has an “Edit this page” button (visible at the top of Figure 2). The button immediately redirects the user to the correct file on GitHub, where they can edit it and submit a pull request with the changes. After the pull request is validated by a CI job and accepted by a curator or technical administrator, the changes automatically appear on the website via a CI/CD pipeline.
RiverBench also collects benchmark run reports and displays them on the website (Figures 2 and 3).
Each benchmark run report is a nanopublication [
Nanopublications with benchmark run reports can be submitted by any researcher with an
ORCID account to the decentralized nanopublication network. A nanopublication template for the web
application Nanodash [
In the live demonstration, we will use the public RiverBench website2 to briefly showcase the main collaboration functionalities of RiverBench. The documentation pages linked in the footnotes contain more details about each functionality.
Reporting benchmark results. We will show how to report the results of benchmarks conducted
using RiverBench3. First, we will fill out the benchmark result form in Nanodash [
Editing documentation and metadata. We will briefly explain the mechanism of editing documentation and the locations of edit buttons on the RiverBench website4. Then, we will demonstrate editing a documentation page and the metadata of a resource (e.g., task or dataset). We will then show to the participants the changes as they appear on the RiverBench website.
Proposing new datasets. We will briefly explain the dataset proposal process 5, and show how to ifll out a dataset proposal form and the metadata file.
Proposing new benchmark tasks. Following a similar structure as with dataset proposals, we will explain the task proposal process6. We will then show how to fill out the form and the metadata file.
In this demo paper, we showcased the collaborative features of RiverBench, an open RDF benchmark suite. The presented four functionalities (reporting benchmark results, editing existing resources, submitting datasets, and proposing benchmark tasks) are supported by comprehensive automation (CI/CD). Additionally, RiverBench itself is a use case of knowledge management techniques, as it extensively uses RDF metadata and Linked Data mechanisms, making this contribution doubly interesting and relevant to the Semantic Web community. The features were designed to be easy to use, open to any contributor, and rely on public and transparent community processes.
To the best of our knowledge, RiverBench is the first RDF benchmark suite to implement collaborative features so widely. We hope that these easy-to-use functionalities will encourage the Semantic Web and Knowledge Graph communities to contribute to RiverBench and join the project as curators, ensuring its long-term relevance and usefulness. The interest of the community ultimately depends on whether the project fulfills an important need or not – in this case, if it useful for benchmarking RDF systems. Currently, RiverBench can be used with nine diverse, scientifically relevant benchmark tasks, but this is expected to be expanded in the future. Based on the early feedback from other researchers, we plan to expand the suite’s infrastructure to support more use cases and tasks. We will also continue improving the suite’s documentation and automation system, to match the changing expectations of the community.
In contrast to the now-defunct Liquid Benchmarks platform [
Online Resources • RiverBench website – the subject of the demo: https://w3id.org/riverbench • RiverBench documentation: https://w3id.org/riverbench/v/dev/documentation • Source code on GitHub: https://github.com/RiverBench