Setting up a local Wikidata SPARQL endpoint is currently technically complex and requires costly hardware resources. In this paper we propose a novel workflow to index Wikidata and setup a local SPARQL endpoint which is easy (simple a docker pull), fast (takes less then 3 hours), cheap (can be executed on a laptop with 16GB of RAM) and eficient (achieves comparable query response times to the oficial endpoint). We achieve this by exploiting a newly open source SPARQL endpoint called the qEndpoint.
also shown in a long standing issue4 to tackle this problem.
One option, suggested also by the community5, is to ofer an easier setup to locally run a query service over Wikidata. This would have mainly two consequences: • the query service at https://query.wikidata.org would have to handle lower pressure allowing a better service overall for all users; • users querying the local setup would not be blocked due to the query limits of the public endpoint6, and should experience less query timeouts since all the local resources would be reserved for them.
While this option seems a viable approach it is technically challenging. The instructions provided by Wikimedia7 to setup a local SPARQL endpoint state that: "it will take about 12 days to get all data imported (into the triple-store), and another 12 days to make the query service catching up the lag". Additionally one needs to add the time to download the dump. Moreover considerable computing resources are necessary8 for the setup, i.e. a server with 100GB of RAM and 16 CPUs. This explains why this solution is not widespread.
In this paper we present an alternative workflow to setup a local SPARQL endpoint over Wikidata. In this version we restrict to a SPARQL endpoint over the truthy statements, but we plan to extend it to the full dump. The workflow allows to setup a local SPARQL endpoint over Wikidata in 3 hours9 that contains data that is not older then 24h and that can run on a laptop with 16GB of RAM. This represents a huge improvement over the existing setup. As a side efect we publish truthy dumps for Wikidata that are more up to date than the once published by Wikidata itself. We achieve this by implementing a new indexing workflow and use a recently created open source triple-store, the qEndpoint.
In this article we are using tools and algorithm, in this section we are describing them.
The qEndpoint is a new triple-store that aims at making RDF datasets queriable at scale with low
hardware requirements. The qEndpoint combines two established Semantic-Web technologies
namely RDF4J[
By taking inspiration on architectures developed in the relational databases world, the qEndpoint combines the advantages of both HDT and RDF4J. It is highly scalable thanks to HDT and supports updates and SPARQL syntax thanks to RDF4J.
The qEndpoint is available as an open source project at https://github.com/the-qa-company/
qEndpoint. It is already used for instance as a part of the infrastructure of the EU Knowledge
Graph[
The compressed RDF dataset format HDT doesn’t supports updates by default, to fix this issue,
2 methods are available, HDTCat[
This method allows to create the UNION of 2 HDT datasets into a new HDT dataset.
HDTCat(ℎ1, ℎ2) = ℎ1 ∪ ℎ2 This method allows to remove triples from an HDT dataset to create a new HDT dataset. We need to give it an HDT ℎ and a bitmap of the size the number of triples of ℎ if a bit = 1, the triple at the index in the HDT should be removed.
HDTDif (ℎ, ) = {ℎ | ℎ ∈ ℎ ∧ = 0}
In the following we describe the workflow that we use to index Wikidata. It makes an intensive use of HDT and some of its related functions. The steps are depicted in Figure 1. 1. We take the truthy dump of Wikidata 16 at a given time 0 and we compress it obtaining 0 . 2. We fetch all the changes since 0 using the Wikimedia Recent Changes API17 and for each of the changed entities we download its current turtle representation18. By concatenating and compressing all these files we create an . 3. We use HDTDif (2.2.2) to eliminate from 0 all triples having as a subject one of the entities we fetched from the recent changes API to obtain 0− . 12https://linkedopendata.eu 13https://app.qanswer.ai 14https://wikidata.qanswer.ai 15https://github.com/rdfhdt/hdt-java/pull/153 16https://dumps.wikimedia.org/wikidatawiki/entities/ 17https://www.mediawiki.org/wiki/API:RecentChanges 18for example for the entity Q1 https://www.wikidata.org/wiki/Special:EntityData/Q1.ttl?flavor=simple 4. Using HDTCat (2.2.1) we join 0− with to obtain an up-to-date HDT.
Finally we compute the corresponding HDT index file. 5. We publish the newly updated Wikidata HDT. 6. We publish a docker image of the qEndpoint that downloads on demand the created
Wikidata HDT.
Note that the steps 1 to 5 are carried out by the publisher. The consumer, i.e. the person that wants to set up the SPARQL endpoint, just needs to download the HDT file and its associated index with the qEndpoint. No other loading time is required. In this case we take advantage of HDT that highly compresses RDF datasets while ofering triple pattern query capabilities.
The steps 1 to 5 are published in the Wiki-changes repository at https://github.com/the-qa-company/ wiki-changes/releases. Step 6 is a docker image that is available at https://hub.docker.com/ r/qacompany/qendpoint-wikidata. It bundles the qEndpoint and downloads at start time the latest HDT index over Wikidata available at https://qanswer-svc4.univ-st-etienne.fr.
First, we show the time that is necessary to generate a fresh index (i.e. steps 1 to 5). To run Wiki-changes, we used a server with 32GB of RAM. The results are presented in the Table 1. We made tests for an 1-day old HDT and a 3-day old HDT. The figures show that once we have the HDT we can get a new up-to-date +1 HDT in less than 18 hours (13 hours for a 1-day old HDT), allowing a maximum diference of 1 day between our HDT and Wikidata.
Next, let’s consider the setup time. On the docker image, the download time of the image of the qEndpoint is only a few minutes and is negligible. Once the container starts, the wikidata index is downloaded from https://qanswer-svc4.univ-st-etienne.fr. The current size of the HDT Wikidata index is 100GB. With a download speed of 10MB/s the endpoint can be setup in less than 3 hours. We reproduced this number. Note that this is just 3 times slower then downloading the full truthy dump19 of wikidata in bz2 format which is 32GB. A consequence of this is also that the space requirement for the local endpoint is as little as 100GB on disk.
In the next step we analyze the performance of the SPARQL endpoint using the oficial query
log of Wikidata[
We have presented an alternative option to set up a local SPARQL endpoint over Wikidata. This solution is quick (3 hours compared to several weeks), low in hardware requirements (only 6GB 19https://dumps.wikimedia.org/wikidatawiki/entities/ 20https://iccl.inf.tu-dresden.de/web/Wikidata_SPARQL_Logs/en 21https://en.wikibooks.org/wiki/SPARQL/SERVICE_-_Label 22-Xmx setting of the JVM 23https://github.com/the-qa-company/qEndpoint/issues/80, https://github.com/eclipse/rdf4j/issues/636 of RAM and 100GB of disk compared to 100GB of RAM and TBs of disk) and fast (it can handle a large amount of real world queries faster than the existing endpoint).
We hope that the easy setup of a local Wikidata endpoint will allow to ofer new types of services around Wikidata that are not bounded by the limits imposed by the oficial Wikidata SPARQL service. We hope also that this setup will move part of the workload away from the public Wikidata endpoint so that it can be hosted in a more decentralized manner.
In future we aim at: • integrating the above workflow and publishing new dumps regularly; • releasing the same setup for the full Wikidata dump; • allow a setup that follows live the changes of Wikidata or Wikibase implementations; • implementing the SPARQL syntax introduced by Blazegraph in order to allow a full compatibility in the current Wikibase ecosystem; • evaluate the endpoint on a larger part of the query log; • take more advantage of the internals of the qEndpoint for analytic queries which is an interesting and exciting direction to further exploit the data published in Wikidata.
Acknowledgment: We would like to thank José M. Giménez-García for the discussions we had while building this work. We would like to thank the HDT community for the amazing work they have done, in particular the creators of HDT and more in particular Javier D. Fernández. We would like the RDF4J community that provides a high quality library with amazing code contributions.