The proliferation of the World Wide Web and the Semantic Web applications has led to an increase in distributed services and datasets. This increase has put the infrastructural load in terms of availability, immutability, and security, and these challenges are being failed by the Linked Open Data (LOD) cloud due to the brittleness of its decentralisation. In this work, we present iLOD: a peerto-peer decentralized storage infrastructure using the InterPlanetary File System (IPFS). iLOD is a dataset sharing system that leverages content-based addressing to support a resilient internet, and can speed up the web by getting nearby copies. In this study, we empirically analyze and evaluate the availability limitations of LOD and propose a distributed system for storing and accessing linked datasets without requiring any centralized server.
The World Wide Web and the Semantic Web are managing a network of distributed
services and datasets in a decentralized way. But because of this decentralization,
sustainability of data is one of the key issues that need to be addressed. An increasing
number of datasets are available as Linked Data, also called the Linked Open Data Cloud.
While the Linked Open Data Cloud does not have a single point of failure, it also has
no mechanism for redundancy so that individual failures gradually decrease the
quality of the cloud. Building a sustainable dataset platform is essential for research, and
in the past we have seen so many linked data projects (like Laundromat [
Decentralized storage (storage of data independently on multiple nodes of the
network in the form of a distributed network) is one of the sustainable solution of datasets
storage. A Peer 2 Peer (P2P) architecture, called the Interplanetary File System (IPFS),
has emerged as a solution to store and access applications, files, websites, and data [
iLOD capitalizes Header, Dictionary, Triples (HDT) format and IPFS technology
to ensure data preservation by storing datasets securely across multiple locations, and
saves in file formats that will likely have the greatest utility in the future [
The rest of the paper is organized as follows: Section 2 discusses the related work in the field of linked data and outlines scope for further research regarding the limitations of linked data. Section 3 describes the proposed iLOD architecture. Finally, the paper concludes by providing a discussion of possible further research. 2
A huge increase in distributed services and datasets resulting in a wide range of
applications across different sectors [
Limitations of Linked Data
One of the central challenges associated with linked data is the increasing obsolescence
of datasets. Polleres et al.[
Datasets Links Full Downloads Other Downloads Examples SPARQL
Available Not Available Percentage the results for the links that are still available are less positive, the vast majority of which are categorized as ‘other downloads’, a category used for indirect links which are often just HTML pages describing the dataset. Direct access to the full data of the datasets is provided by only 25.0% of datasets in the cloud and of those nearly half are no longer available. Similarly, for ‘examples’ which represent links to a single URL published as linked data (i.e., using content negotiation), we see that most of these resources are unavailable as is also the case for SPARQL endpoints. This shows some of the challenges with publishing data using linked data and maintaining these endpoints for a long time.
Table 1 also presents the availability and total amount of links organized by media
type. Most of the major RDF datatypes available in the cloud are Turtle (363 links)
followed by RDF/XML (358 links), other formats such as N-Triples (55), JSON-LD
(24) and TriG (4) are much less popular, although it is noteworthy that N-Triples are
likely much more popular, but as the MIME type was only approved recently and
NTriples are frequently compressed, data in this format may be listed under other
media types, such as text/plain or application/octet-stream. Further, there
are currently no links indicating the use of HDT. We also see that the most common
method of publishing data2 is ZIP with 95.7%. This fits with the authors’ experience
in running numerous linked data sites; it is hard to maintain these datasets over the
long term, when combined with complex methodologies such as content negotiation
2 Ignoring formats used for documentation such as HTML and PDF
and SPARQL querying, while a simple dump of the data is the best way to ensure that
there is long-term availability of datasets. Likewise, HDT takes much less time than
traditional techniques to download and start querying a dataset [
Up to this point, our analysis has primarily agreed with Polleres [
In this section, we describe the IPFS solution for LOD datasets. The rationale for this idea is derived from the Table 1. iLOD mainly comprises of four parts: Data acquisition, format conversion, IPFS addition, and cloud generation (see Figure 1).
Data Acquisition: To allow for maximum flexibility in combining and reusing LOD, we consider the LOD cloud as a dataset information provider, and the LOD Laundromat dump as a data provider. Overlapping datasets are collected. This configuration is for demo only, but any dataset can be used to add in iLOD.
Format Conversion: iLOD converts all collected datasets into HDT format. This compression makes datasets more smaller, while maintaining (and even improving) search and browse operations without prior decompression.
IPFS Addition: The HDT-converted datasets are then added to the IPFS system to generate a cryptographic hash content identifier (CID) for each dataset. CID label is a content-based address. Once CID has been generated, the dataset is available on the IPFS network and ready to share with everyone. Anyone having the CID can get content without relying on the (temporary) location of the content. Moreover, this can be used as the target of links by using IPFS URLs such as:
A potential drawback of using these IDs is that content negotiation cannot be implemented. This could easily be fixed by providing good metadata in the HDT file to provide alternative versions of the data and backlinks to previous versions that can be easily compiled to provide the most up-to-date version of a dataset. The good thing about IPFS is that if the same dataset is added to two different IPFS nodes, using the same settings, it will produce exactly the same CID, so redundancy issue can also be resolved. A (distributed) index of available datasets can be used to track all datasets and hence find the most recent version.
iLOD Cloud: Generated CIDs are then added to the iLOD cloud. The iLOD cloud 3 is a loosely coupled collection of IPFS datasets. As the iLOD cloud is built on top of the LOD cloud, all the advantages and information of LOD are also available for iLOD. Just like any P2P application, iLOD requires active participation in the network in the form of ‘pinning’ (mirroring) the content, however it is robust to the disappearance of the original data publisher in the network unlike the current LOD cloud. 4
We have introduced iLOD – an IPFS-based Linked Open Data that allows users to easily add new datasets or download already added datasets. We suggest extending the linked data principles to require the publishing of linked data as HDT, but flexibility can be added if the data is provided with its metadata in a single file following a standard. We also propose a revision of the second principles from “Use HTTP URIs so that people can look up those names” to “use stable, content-based identifiers such as IPFS/CID URLs to make links so that people can find the data forever.” With this, we hope to provide a sustainable platform for LOD storage and sharing. In future, we want to extend this work by automatically finding metadata of datasets and their relations with each other in Laundromat dump and then add these datasets to iLOD cloud.