In March 2013, the W3C recommended SPARQL 1.1 to retrieve and manipulate decentralized RDF data. Real-world usage requires advanced features of SPARQL 1.1. recommendations As these are not consistently implemented, we propose a test framework named TFT (Tests for Triple stores) to test the interoperability of the SPARQL endpoint of RDF database systems. This framework can execute the W3C's SPARQL 1.1 test suite and also its own tests of interoperability. To help the developers and end-users of RDF databases, we perform daily tests on Jena-Fuseki, Marmotta-KiWistore, 4Store and three other commercial databases. With these tests, we have built a scoring system named SPARQLScore and share our results on the website http://sparqlscore.com.
The current W3C recommendation SPARQL 1.1 has been published in its
nal form in May 2013 [
Thus, predicting beforehand the support of a particular SPARQL 1.1 feature in a given RDF database was impossible. This is generally damaging the deployment of the Semantic Web, and has particularly pernicious consequences in scienti c research ecosystems.
In the CDS (Center for Data Science) project of Paris-Saclay University, we develop an integrated framework that o ers a seamless facility to run and exploit exhaustive testing of RDF databases, in order to help our scienti c communities to choose the best solution to share their data. Our panel is broad: large and well-organized communities such as High Energy Physics (CERN experiments) as well as local communities that just discover the need to share beyond shortlived experiments, and many more; it includes both hard and soft science.
Our current TFT work ow automatically compiles, deploys and tests every night several hand-picked RDF databases from their sources as well as one SPARQL endpoint o ered by a software vendor. It maintains a database of test results accessible from a web interface. The work ow will shortly be integrated within a platform as a service (PaaS), where TFT will be used to evaluate the conformity of a virtual image hosting an open source RDF database to the latest SPARQL standards, thus providing the scienti c end users with critical information for a better informed choice of database based on their needs (performance, support for a particular ontology, etc.), including SPARQL federated queries. The vendors will also be able to propose virtual machines including their own RDF database system, which will then be automatically evaluated using the TFT software before being proposed to researchers.
The Semantic Web, or Web of Data, aims among other things to share readable information between humans and machines. When this exchange will be possible, new machines will be born to help the humans to use all the information on the Web. This huge amount of information is already unusable by humans but the majority of the machines are also unable of handling it alone.
The machines on the Web become specialized: collector, calculator, semantic parser, databases, etc. The availability of APIs with the WebService technology was the rst response to the need to communicate between machines. Unfortunately, there are as many APIs as developers. This heterogeneity makes impossible the implementation of autonomous agents able to discover and consume the Web data, as it had set a simple API and a unique protocol. Enabling such agents is the aim of SPARQL, by making them capable to discover the data across the Web without downloading them beforehand. This is also a major issue for the Web of Things, where every object becomes a potential Web Agent.
Lack of interoperability causes two complications that are critical to widespread adoption of RDF by large and organized scienti c communities such as HEP (High Energy Physics): migration between databases and their updgrades and the development of agents, as experienced manpower is scarce.
Total interoperability might still be a long way to go. Should we wait until it happens? Instead, a medium term strategy can take into account the fact that, in practice, the end-users could cope with some limitations. However, their tradeo s are di erent, thus a rst-order requirement is to be able to precisely assess the strengths and aws of databases with respect to interoperability. The tool for evaluating interoperability did not exist. TFT answers this critical need.
Interoperability is not enough for scienti c communities. The most advanced ones want to use the latest database technology (inferences, velocity, clustering and so on). These innovations are rarely available in the stable versions of databases before several months or even several years. The unstable versions are often available for free download and researchers can install these latest versions very quickly with tools like Git. Moreover, for the small communities, the compromise between compliance to standards and cutting-edge performance is often arbitrated more or less blindly in favor of the latter. The TFT software provides a simple solution to make a better informed decision, creating an incentive for selecting the more interoperable technology within the user requirements. 2.3
For a Chief Information O cer (CIO) in the academic world providing services to multiple small and poorly organized scienti c communities, the condition of interoperability is not enough to deploy a software in an information system. The CIOs have strong Quality of Service constraints (QoS). The solution of CIO is to o er an IaaS (Infrastructure as a Service, typically a local cloud). With this IaaS, the researcher can create or disappear a virtual machine in a few clicks, and can install her preferred tools without bothering with QoS, security and interoperability. After evaluation of the research results by the peers, the resources may disappear fairly quickly because the corresponding data are still rarely integrated in a long term archiving plan.
These careless methods are doomed. The scienti c agencies are enforcing the
requirement to linking data to results, with reproducibility as the ultimate goal.
Nobody can replace the researchers to save their work and share their ndings,
with mechanisms such as the Digital Object Identi er (DOI) System [
The TFT software certi es the last version of RDF database system using a continuous delivery work ow. By providing seamless choice of the last best interoperable databases, our innovation facilitates data sharing within and across scienti c communities. Beyond selection, our PaaS contributes to the advent of reproducible science. 3 3.1
The TFT has 4 parts: 1) upload the benchmarks into our RDF database, 2) run the tests, 3) compute a score for each database systems, and 4) share the results in RDF via SPARQL.
End user
Deliveryteam
Currently TFT o ers a score on interoperability of software and also provide
a RDF database of detailed test results. In the near future, these results can
power tools in the cloud that will facilitate the provision of solution of latest
generation databases for the researchers and will be maintained by CIOs, by
ensuring interoperability of data, and will facilitate their work of preserving
data by being able to simply migrate data from one system to another. TFT can
also be integrated into continuous integration environments of database editors,
in order to improve their products and the CIOs can check the RDF database
system in their environments. Fig. 1 summarizes the work ow. After a new
release of a given RDF database, software is made available in the form of a
virtual appliance, the image is run on a validation cloud and a set of tests is
performed over this new database instance. According to the results of the tests,
the validation can either (a) fail and a feedback is provided to the appliances
publisher or (b) user-de ned tests are run to validate their speci c needs.
Upload the tests. For now, there are two collections of tests: the SPARQL 1.1 test
suite (453 tests) [
] ; qt : s e r v i c e D a t a [ qt : endpoint <http : / / example2 . org / sparql > ; qt : data <b d i i . t t l > ] ; mf : r e s u l t <q10 . srx> . [ SERVICE ] endpoint [ " http : / / example . org / s p a r q l " ] = " http : / / o1 . in2p3 . f r / s p a r q l /" endpoint [ " http : / / example1 . org / s p a r q l " ] = " http : / / o2 . in2p3 . f r / s p a r q l /" endpoint [ " http : / / example2 . org / s p a r q l " ] = " http : / / o3 . in2p3 . f r / s p a r q l /" o . / j u n i t n r $fBUILD URLg n softwareName=Fuseki n softwareDescribeTag=v$fVERSIONFUSEKIg n s o f t w a r e D e s c r i b e="$fBUILD TAGg#$fFILEFUSEKIg" the second remote endpoint. During the tests, TFT will replace the URIs of the remote endpoints with the URI contained in the le config.ini (Fig. 3).
Pass the tests. We use Jenkins, a continuous integration server, because the database systems are often Open Source and in a Git repository. Fig. 4 shows an example of a script execution by our continuous integration server. It follows the same work ow for each test: 1) delete all data from the main test database and the remote test database(s); 2) load initial data to de ne the initial state in the main database and the remote database(s); 3) run the tests in the main database; in case of federated queries, the main database is responsible for contacting the remote ones (this is the normal behavior of federated queries) ; 4) Monitor the response to the test and/or control the nal state obtained in the databases. After the tests have been run, the script tft saves and shares the results.
Compute a score. Choosing a particular weight for each of our 459 tests would be highly debatable. We calculate a simple global score for each RDF database system: one point is given for each passed test. The script tft-score calculates this score and shares these scores and the results of tests. After the compliance tests have been run, we share three results with various actors
With the editors. TFT creates a report in JUnit format compliant with the
Jenkins software (Fig. 5, left). Jenkins can check the last push in the Git
repository about a software and can give a feedback to developers in real-time. If a
software editor integrates TFT in its Jenkins server, he will also be able to reject
automatically the last delivery if a test shows a regression of interoperability. An
example of tests is available in the project TFT-tests on GitHub [
With the machine. After the compliance tests have been run, TFT generates
two reports: one in JUnit format for our own Jenkins server and one in
RDF/EARL (Evaluation and Report Language) format [
With the end-users The SparqlScore.com website, Fig. 5 (right), illustrates
the reuse of the test results and database scores. In order to improve the user's
experience of the website and to relieve our database, we use the Smarty [
We integrate the linked data technologies where the input are the tests in the turtle format with the ontology de ned by the SPARQL 1.1 WG; the output is a RDF database that is fed by a SPARQL Update query after each test.
This design o ers the possibility to write quickly a new test and everybody
can propose a new test or fork the tests via a project in the GitHub's Service
[
The TFT software is under a Creative Commons Attribution-ShareAlike 4.0
International License. The aim of this license is to share the same software to
test and compare objectively the databases on the market. TFT and TFT-tests
(the collections of tests) are available via their repositories [
The SparqlScore software is also available via its repository [
Benchmarking without testing the protocol is insu cient The SPARQL update protocol is not identical across databases. Developing a Web agent with SPARQL without knowing the exact server software is quite di cult. The reason is simple: the protocol concerning update queries is fuzzy in the SPARQL recommendation thus each database implements a di erent avor of the protocol. TFT can test ve RDF databases because we have to implement the speci cities of each database in order to execute the same queries. An open benchmark is possible and can help to converge Very quickly after the launch of the sparqlscore.com website, four vendors contacted us to include their software in our tests and three accepted to open their results. Three vendors have a speci cally set-up a SPARQL endpoint for our tests. The editors started to discuss how to interpret the recommendation and several xed some interoperability problems.
The SPARQL 1.1 is a recommendation but not the tests
The o cial test suite is a great job as a starting point. Each di erence in the
result of a SPARQL query in an RDF database is an obstacle to the deployment
of Linked Data in public institutions or a simple company. But interoperability is
not an option in the Linked Data, it's the rst aim. Moreover, a lot people want
use the Linked Data's technologies and the classical access control problems are
resolved separately by the di erent editors. The editors have to diverge from the
recommendation in order to resolve the security needs of their customers.
Help to create a really interoperable ecosystem ?
The W3C has launched the Test the Web Forward[
We hope TFT can be useful for the possible evolution of the \Test the Web Forward" initiative to create a really interoperable ecosystem for the Linked Data.
This work has been partially funded by the TIMCO project, by the Paris-Saclay Center for Data Science (funded by the IDEX Paris-Saclay, ANR-11-IDEX-000302), and by France Grilles.